2019-12-21T15:33:05.611 INFO:root:teuthology version: 1.0.0-744c324 2019-12-21T15:33:05.611 DEBUG:teuthology.run:Teuthology command: teuthology /tmp/teuthology-worker.YADvkl.tmp -- --verbose --lock --description rados/standalone/{supported-random-distro$/{rhel_8.yaml} workloads/scrub.yaml} --name sage-2019-12-21_15:05:37-rados-wip-sage-testing-2019-12-20-2043-distro-basic-smithi --block --owner scheduled_sage@teuthology --archive /home/teuthworker/archive/sage-2019-12-21_15:05:37-rados-wip-sage-testing-2019-12-20-2043-distro-basic-smithi/4624231 2019-12-21T15:33:05.635 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2019-12-21T15:33:05.872 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/sage-2019-12-21_15:05:37-rados-wip-sage-testing-2019-12-20-2043-distro-basic-smithi/4624231 branch: wip-sage-testing-2019-12-20-2043 description: rados/standalone/{supported-random-distro$/{rhel_8.yaml} workloads/scrub.yaml} email: null first_in_suite: false job_id: '4624231' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: sage-2019-12-21_15:05:37-rados-wip-sage-testing-2019-12-20-2043-distro-basic-smithi nuke-on-error: true openstack: - volumes: count: 3 size: 10 os_type: rhel os_version: '8.0' overrides: admin_socket: branch: wip-sage-testing-2019-12-20-2043 ceph: conf: mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug filestore: 20 debug journal: 20 debug ms: 1 debug osd: 25 log-whitelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: c6e9f8f9e7516809792ffb509ef4a897f5c2f45e ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 install: ceph: sha1: c6e9f8f9e7516809792ffb509ef4a897f5c2f45e workunit: branch: wip-sage-testing-2019-12-21-0902 sha1: 2b368c3feb5f798e3a142f2a972094a812d3d958 owner: scheduled_sage@teuthology priority: 90 repo: git://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 sha1: c6e9f8f9e7516809792ffb509ef4a897f5c2f45e suite: rados suite_branch: wip-sage-testing-2019-12-21-0902 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_wip-sage-testing-2019-12-21-0902/qa suite_relpath: qa suite_repo: git://git.ceph.com/ceph-ci.git suite_sha1: 2b368c3feb5f798e3a142f2a972094a812d3d958 tasks: - install: null - workunit: basedir: qa/standalone clients: all: - scrub teuthology_branch: master tube: smithi verbose: false worker_log: /home/teuthworker/archive/worker_logs/worker.smithi.2507 2019-12-21T15:33:05.873 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_wip-sage-testing-2019-12-21-0902/qa; will attempt to use it 2019-12-21T15:33:05.873 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_wip-sage-testing-2019-12-21-0902/qa/tasks 2019-12-21T15:33:05.874 INFO:teuthology.run_tasks:Running task internal.check_packages... 2019-12-21T15:33:05.876 INFO:teuthology.task.internal:Checking packages... 2019-12-21T15:33:05.890 INFO:teuthology.task.internal:Checking packages for os_type 'rhel', flavor 'basic' and ceph hash 'c6e9f8f9e7516809792ffb509ef4a897f5c2f45e' 2019-12-21T15:33:05.890 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2019-12-21T15:33:05.890 INFO:teuthology.packaging:ref: None 2019-12-21T15:33:05.890 INFO:teuthology.packaging:tag: None 2019-12-21T15:33:05.891 INFO:teuthology.packaging:branch: wip-sage-testing-2019-12-20-2043 2019-12-21T15:33:05.891 INFO:teuthology.packaging:sha1: c6e9f8f9e7516809792ffb509ef4a897f5c2f45e 2019-12-21T15:33:05.891 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-sage-testing-2019-12-20-2043 2019-12-21T15:33:06.049 INFO:teuthology.task.internal:Found packages for ceph version 15.0.0-8795.gc6e9f8f 2019-12-21T15:33:06.050 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2019-12-21T15:33:06.059 INFO:teuthology.task.internal:no buildpackages task found 2019-12-21T15:33:06.059 INFO:teuthology.run_tasks:Running task internal.lock_machines... 2019-12-21T15:33:06.074 INFO:teuthology.task.internal.lock_machines:Locking machines... 2019-12-21T15:33:06.074 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2019-12-21T15:33:06.134 DEBUG:teuthology.lock.ops:lock_many request: {'count': 1, 'locked_by': 'scheduled_sage@teuthology', 'description': '/home/teuthworker/archive/sage-2019-12-21_15:05:37-rados-wip-sage-testing-2019-12-20-2043-distro-basic-smithi/4624231', 'machine_type': 'smithi'} 2019-12-21T15:33:06.151 DEBUG:teuthology.lock.ops:locked ubuntu@smithi172.front.sepia.ceph.com 2019-12-21T15:33:06.180 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=, signals=[15]) 2019-12-21T15:33:06.378 INFO:teuthology.provision.fog.smithi172:Scheduling deploy of rhel 8.0 2019-12-21T15:33:06.682 INFO:teuthology.orchestra.console:Power off smithi172 2019-12-21T15:33:06.682 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi172.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2019-12-21T15:33:06.704 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi172.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2019-12-21T15:33:10.744 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi172.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2019-12-21T15:33:14.856 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi172.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2019-12-21T15:33:15.067 INFO:teuthology.orchestra.console:Power off for smithi172 completed 2019-12-21T15:33:15.168 INFO:teuthology.orchestra.console:Power on smithi172 2019-12-21T15:33:15.168 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi172.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2019-12-21T15:33:15.190 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi172.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2019-12-21T15:33:19.212 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi172.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2019-12-21T15:33:23.328 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi172.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2019-12-21T15:33:23.538 INFO:teuthology.orchestra.console:Power on for smithi172 completed 2019-12-21T15:33:23.639 INFO:teuthology.provision.fog.smithi172:Waiting for deploy to finish 2019-12-21T15:35:54.624 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi172.front.sepia.ceph.com', 'timeout': 60} 2019-12-21T15:35:54.723 ERROR:teuthology.orchestra.connection:Error connecting to smithi172.front.sepia.ceph.com Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/connection.py", line 106, in connect ssh.connect(**connect_args) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/paramiko/client.py", line 446, in connect passphrase, File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/paramiko/client.py", line 764, in _auth raise saved_exception AuthenticationException: Authentication failed. 2019-12-21T15:37:01.727 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi172.front.sepia.ceph.com', 'timeout': 60} 2019-12-21T15:38:07.731 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi172.front.sepia.ceph.com', 'timeout': 60} 2019-12-21T15:38:07.953 INFO:teuthology.orchestra.run.smithi172:> hostname 2019-12-21T15:38:08.251 INFO:teuthology.orchestra.run.smithi172.stdout:smithi172.front.sepia.ceph.com 2019-12-21T15:38:08.251 INFO:teuthology.orchestra.run.smithi172:> grep smithi172.front.sepia.ceph.com /etc/hosts 2019-12-21T15:38:08.265 INFO:teuthology.orchestra.run.smithi172.stdout:172.21.15.172 smithi172.front.sepia.ceph.com smithi172 2019-12-21T15:38:08.266 INFO:teuthology.orchestra.run.smithi172:> sudo hostname smithi172 2019-12-21T15:38:08.377 INFO:teuthology.orchestra.run.smithi172:> sudo sed -i -e 's/smithi172.front.sepia.ceph.com/smithi172/g' /etc/hosts 2019-12-21T15:38:08.404 INFO:teuthology.orchestra.run.smithi172:> sudo sed -i -e 's/172.21.15.172/172.21.15.172/g' /etc/hosts 2019-12-21T15:38:08.471 INFO:teuthology.orchestra.run.smithi172:> sudo sed -i -e 's/smithi172.front.sepia.ceph.com/smithi172/g' /etc/hostname 2019-12-21T15:38:08.539 INFO:teuthology.orchestra.run.smithi172:> sudo hostname smithi172 2019-12-21T15:38:08.605 INFO:teuthology.provision.fog.smithi172:Deploy complete! 2019-12-21T15:38:08.606 DEBUG:teuthology.parallel:result is None 2019-12-21T15:38:08.711 INFO:teuthology.lock.ops:Checking smithi172.front.sepia.ceph.com 2019-12-21T15:38:08.712 INFO:teuthology.lock.ops:New key found. Updating... 2019-12-21T15:38:08.729 INFO:teuthology.lock.ops:Updating [smithi172.front.sepia.ceph.com]: set os type and version on server 2019-12-21T15:38:08.729 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2019-12-21T15:38:08.730 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi172.front.sepia.ceph.com', 'timeout': 60} 2019-12-21T15:38:08.796 INFO:teuthology.orchestra.run.smithi172:> true 2019-12-21T15:38:08.959 INFO:teuthology.orchestra.run.smithi172:> uname -m 2019-12-21T15:38:09.013 INFO:teuthology.orchestra.run.smithi172.stdout:x86_64 2019-12-21T15:38:09.013 INFO:teuthology.orchestra.run.smithi172:> cat /etc/os-release 2019-12-21T15:38:09.066 INFO:teuthology.orchestra.run.smithi172.stdout:NAME="Red Hat Enterprise Linux" 2019-12-21T15:38:09.066 INFO:teuthology.orchestra.run.smithi172.stdout:VERSION="8.0 (Ootpa)" 2019-12-21T15:38:09.066 INFO:teuthology.orchestra.run.smithi172.stdout:ID="rhel" 2019-12-21T15:38:09.067 INFO:teuthology.orchestra.run.smithi172.stdout:ID_LIKE="fedora" 2019-12-21T15:38:09.067 INFO:teuthology.orchestra.run.smithi172.stdout:VERSION_ID="8.0" 2019-12-21T15:38:09.067 INFO:teuthology.orchestra.run.smithi172.stdout:PLATFORM_ID="platform:el8" 2019-12-21T15:38:09.067 INFO:teuthology.orchestra.run.smithi172.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.0 (Ootpa)" 2019-12-21T15:38:09.067 INFO:teuthology.orchestra.run.smithi172.stdout:ANSI_COLOR="0;31" 2019-12-21T15:38:09.067 INFO:teuthology.orchestra.run.smithi172.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8.0:GA" 2019-12-21T15:38:09.068 INFO:teuthology.orchestra.run.smithi172.stdout:HOME_URL="https://www.redhat.com/" 2019-12-21T15:38:09.068 INFO:teuthology.orchestra.run.smithi172.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2019-12-21T15:38:09.068 INFO:teuthology.orchestra.run.smithi172.stdout: 2019-12-21T15:38:09.068 INFO:teuthology.orchestra.run.smithi172.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2019-12-21T15:38:09.068 INFO:teuthology.orchestra.run.smithi172.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.0 2019-12-21T15:38:09.068 INFO:teuthology.orchestra.run.smithi172.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2019-12-21T15:38:09.068 INFO:teuthology.orchestra.run.smithi172.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.0" 2019-12-21T15:38:09.069 INFO:teuthology.lock.ops:Updating smithi172.front.sepia.ceph.com on lock server 2019-12-21T15:38:09.092 INFO:teuthology.task.internal.lock_machines:1 smithi machines locked this try, 1/1 locked so far 2019-12-21T15:38:09.108 INFO:teuthology.task.internal.lock_machines:Locked targets: smithi172.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCYbVb+3NyuAFbEmX4BN76gGmrxrZiD+8kmItwHX485P2PGH00HpUJkQ+1oxqOqa+KIBLmLudvvXuHqkup9cUP9zM2fHaDvOdXxsGTZvSeQJs7U9ZnukKzj+yAQFtZu+2fBBFfirrL62THQfenbAo22RcRt2xSUEMCN0/kJ6a9oyag1HJsbqWoJudcEW5XXxbtr+dwPdZLtC9oz1irVVZAfzFdsnoIKTwxEkbujOAJG1e5FvsOIJwnh6wcJg/4FOtzNUnG6HWvyDH5wykwqfwNxGy0iDuP5gFo+WlB1RVxXRf5cU2HViDjH7yJAOsYCgixdldirjTGH8PVpitkQnlzP 2019-12-21T15:38:09.109 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2019-12-21T15:38:09.155 INFO:teuthology.run_tasks:Running task internal.save_config... 2019-12-21T15:38:09.165 INFO:teuthology.task.internal:Saving configuration 2019-12-21T15:38:09.177 INFO:teuthology.run_tasks:Running task internal.check_lock... 2019-12-21T15:38:09.221 INFO:teuthology.task.internal.check_lock:Checking locks... 2019-12-21T15:38:09.239 DEBUG:teuthology.task.internal.check_lock:machine status is {u'is_vm': False, u'locked': True, u'locked_since': u'2019-12-21 15:33:06.143646', u'locked_by': u'scheduled_sage@teuthology', u'up': True, u'mac_address': None, u'name': u'smithi172.front.sepia.ceph.com', u'os_version': u'8.0', u'machine_type': u'smithi', u'vm_host': None, u'os_type': u'rhel', u'arch': u'x86_64', u'ssh_pub_key': u'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIzAkdNgQsZVG6gFo3yBOnTpyaixuhfHVJCu5hf9hI2O7e4VJgzt9srRvEU/mvawU8FBa0Ejxmhfvqmq//0l6M0=', u'description': u'/home/teuthworker/archive/sage-2019-12-21_15:05:37-rados-wip-sage-testing-2019-12-20-2043-distro-basic-smithi/4624231'} 2019-12-21T15:38:09.239 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2019-12-21T15:38:09.249 INFO:teuthology.task.internal:roles: ubuntu@smithi172.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2019-12-21T15:38:09.250 INFO:teuthology.run_tasks:Running task console_log... 2019-12-21T15:38:09.293 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=, signals=[15]) 2019-12-21T15:38:09.293 INFO:teuthology.run_tasks:Running task internal.connect... 2019-12-21T15:38:09.304 INFO:teuthology.task.internal:Opening connections... 2019-12-21T15:38:09.304 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi172.front.sepia.ceph.com 2019-12-21T15:38:09.305 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi172.front.sepia.ceph.com', 'timeout': 60} 2019-12-21T15:38:09.373 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2019-12-21T15:38:09.385 INFO:teuthology.orchestra.run.smithi172:> uname -m 2019-12-21T15:38:09.535 INFO:teuthology.orchestra.run.smithi172.stdout:x86_64 2019-12-21T15:38:09.535 INFO:teuthology.orchestra.run.smithi172:> cat /etc/os-release 2019-12-21T15:38:09.588 INFO:teuthology.orchestra.run.smithi172.stdout:NAME="Red Hat Enterprise Linux" 2019-12-21T15:38:09.589 INFO:teuthology.orchestra.run.smithi172.stdout:VERSION="8.0 (Ootpa)" 2019-12-21T15:38:09.589 INFO:teuthology.orchestra.run.smithi172.stdout:ID="rhel" 2019-12-21T15:38:09.589 INFO:teuthology.orchestra.run.smithi172.stdout:ID_LIKE="fedora" 2019-12-21T15:38:09.589 INFO:teuthology.orchestra.run.smithi172.stdout:VERSION_ID="8.0" 2019-12-21T15:38:09.589 INFO:teuthology.orchestra.run.smithi172.stdout:PLATFORM_ID="platform:el8" 2019-12-21T15:38:09.589 INFO:teuthology.orchestra.run.smithi172.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.0 (Ootpa)" 2019-12-21T15:38:09.589 INFO:teuthology.orchestra.run.smithi172.stdout:ANSI_COLOR="0;31" 2019-12-21T15:38:09.590 INFO:teuthology.orchestra.run.smithi172.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8.0:GA" 2019-12-21T15:38:09.590 INFO:teuthology.orchestra.run.smithi172.stdout:HOME_URL="https://www.redhat.com/" 2019-12-21T15:38:09.590 INFO:teuthology.orchestra.run.smithi172.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2019-12-21T15:38:09.590 INFO:teuthology.orchestra.run.smithi172.stdout: 2019-12-21T15:38:09.590 INFO:teuthology.orchestra.run.smithi172.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2019-12-21T15:38:09.590 INFO:teuthology.orchestra.run.smithi172.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.0 2019-12-21T15:38:09.590 INFO:teuthology.orchestra.run.smithi172.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2019-12-21T15:38:09.591 INFO:teuthology.orchestra.run.smithi172.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.0" 2019-12-21T15:38:09.591 INFO:teuthology.lock.ops:Updating smithi172.front.sepia.ceph.com on lock server 2019-12-21T15:38:09.605 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2019-12-21T15:38:09.624 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2019-12-21T15:38:09.633 INFO:teuthology.task.internal:Checking for old test directory... 2019-12-21T15:38:09.634 INFO:teuthology.orchestra.run.smithi172:> test '!' -e /home/ubuntu/cephtest 2019-12-21T15:38:09.647 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2019-12-21T15:38:09.657 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2019-12-21T15:38:09.658 INFO:teuthology.orchestra.run.smithi172:> test -z $(ls -A /var/lib/ceph) 2019-12-21T15:38:09.704 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2019-12-21T15:38:09.737 INFO:teuthology.run_tasks:Running task kernel... 2019-12-21T15:38:09.757 DEBUG:teuthology.task.kernel:normalized config {'mon.a': {'sha1': 'distro', 'kdb': True}, 'osd.2': {'sha1': 'distro', 'kdb': True}, 'osd.1': {'sha1': 'distro', 'kdb': True}, 'osd.0': {'sha1': 'distro', 'kdb': True}, 'client.0': {'sha1': 'distro', 'kdb': True}, 'mgr.x': {'sha1': 'distro', 'kdb': True}} 2019-12-21T15:38:09.757 INFO:teuthology.task.kernel:config {'mon.a': {'sha1': 'distro', 'kdb': True}}, timeout 300 2019-12-21T15:38:09.757 INFO:teuthology.orchestra.run.smithi172:> uname -r 2019-12-21T15:38:09.771 INFO:teuthology.orchestra.run.smithi172.stdout:4.18.0-80.11.2.el8_0.x86_64 2019-12-21T15:38:09.771 INFO:teuthology.task.kernel:Running kernel on smithi172: 4.18.0-80.11.2.el8_0.x86_64 2019-12-21T15:38:09.771 INFO:teuthology.orchestra.run.smithi172:> sudo yum install -y kernel 2019-12-21T15:38:11.367 INFO:teuthology.orchestra.run.smithi172.stdout:Updating Subscription Management repositories. 2019-12-21T15:38:11.369 INFO:teuthology.orchestra.run.smithi172.stdout:Unable to read consumer identity 2019-12-21T15:38:11.531 INFO:teuthology.orchestra.run.smithi172.stdout:This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register. 2019-12-21T15:38:12.693 INFO:teuthology.orchestra.run.smithi172.stdout:Extra Packages for Enterprise Linux 8 - x86_64 59 kB/s | 19 kB 00:00 2019-12-21T15:38:13.494 INFO:teuthology.orchestra.run.smithi172.stdout:Extra Packages for Enterprise Linux 8 - x86_64 5.4 MB/s | 4.3 MB 00:00 2019-12-21T15:38:14.384 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:00:01 ago on Sat 21 Dec 2019 03:38:13 PM UTC. 2019-12-21T15:38:14.676 INFO:teuthology.orchestra.run.smithi172.stdout:Package kernel-4.18.0-80.el8.x86_64 is already installed. 2019-12-21T15:38:14.677 INFO:teuthology.orchestra.run.smithi172.stdout:Package kernel-4.18.0-80.11.2.el8_0.x86_64 is already installed. 2019-12-21T15:38:14.752 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2019-12-21T15:38:14.753 INFO:teuthology.orchestra.run.smithi172.stdout:Nothing to do. 2019-12-21T15:38:14.753 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2019-12-21T15:38:14.900 INFO:teuthology.orchestra.run.smithi172:> echo no | sudo yum reinstall kernel || true 2019-12-21T15:38:15.223 INFO:teuthology.orchestra.run.smithi172.stdout:Updating Subscription Management repositories. 2019-12-21T15:38:15.223 INFO:teuthology.orchestra.run.smithi172.stdout:Unable to read consumer identity 2019-12-21T15:38:15.277 INFO:teuthology.orchestra.run.smithi172.stdout:This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register. 2019-12-21T15:38:15.391 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:00:02 ago on Sat 21 Dec 2019 03:38:13 PM UTC. 2019-12-21T15:38:15.503 INFO:teuthology.orchestra.run.smithi172.stdout:Installed package kernel-4.18.0-80.el8.x86_64 (from anaconda) not available. 2019-12-21T15:38:15.503 INFO:teuthology.orchestra.run.smithi172.stdout:Installed package kernel-4.18.0-80.11.2.el8_0.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2019-12-21T15:38:15.554 INFO:teuthology.orchestra.run.smithi172.stderr:Error: No packages marked for reinstall. 2019-12-21T15:38:15.614 INFO:teuthology.orchestra.run.smithi172:> sudo yum reinstall -y kernel || true 2019-12-21T15:38:15.940 INFO:teuthology.orchestra.run.smithi172.stdout:Updating Subscription Management repositories. 2019-12-21T15:38:15.940 INFO:teuthology.orchestra.run.smithi172.stdout:Unable to read consumer identity 2019-12-21T15:38:15.994 INFO:teuthology.orchestra.run.smithi172.stdout:This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register. 2019-12-21T15:38:16.108 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:00:03 ago on Sat 21 Dec 2019 03:38:13 PM UTC. 2019-12-21T15:38:16.210 INFO:teuthology.orchestra.run.smithi172.stdout:Installed package kernel-4.18.0-80.el8.x86_64 (from anaconda) not available. 2019-12-21T15:38:16.211 INFO:teuthology.orchestra.run.smithi172.stdout:Installed package kernel-4.18.0-80.11.2.el8_0.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2019-12-21T15:38:16.254 INFO:teuthology.orchestra.run.smithi172.stderr:Error: No packages marked for reinstall. 2019-12-21T15:38:16.312 INFO:teuthology.orchestra.run.smithi172:> rpm -q kernel --last | head -n 1 2019-12-21T15:38:16.345 INFO:teuthology.orchestra.run.smithi172.stdout:kernel-4.18.0-80.11.2.el8_0.x86_64 Wed 11 Dec 2019 02:25:38 PM UTC 2019-12-21T15:38:16.346 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-80.11.2.el8_0.x86_64 2019-12-21T15:38:16.346 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2019-12-21T15:38:16.346 INFO:teuthology.orchestra.run.smithi172:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2019-12-21T15:38:16.424 INFO:teuthology.orchestra.run.smithi172.stdout:ttyS1 2019-12-21T15:38:16.437 INFO:teuthology.run_tasks:Running task internal.base... 2019-12-21T15:38:16.448 INFO:teuthology.task.internal:Creating test directory... 2019-12-21T15:38:16.449 INFO:teuthology.orchestra.run.smithi172:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2019-12-21T15:38:16.491 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2019-12-21T15:38:16.502 INFO:teuthology.run_tasks:Running task internal.archive... 2019-12-21T15:38:16.516 INFO:teuthology.task.internal:Creating archive directory... 2019-12-21T15:38:16.516 INFO:teuthology.orchestra.run.smithi172:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2019-12-21T15:38:16.555 INFO:teuthology.run_tasks:Running task internal.coredump... 2019-12-21T15:38:16.567 INFO:teuthology.task.internal:Enabling coredump saving... 2019-12-21T15:38:16.567 INFO:teuthology.orchestra.run.smithi172:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2019-12-21T15:38:16.623 INFO:teuthology.orchestra.run.smithi172.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2019-12-21T15:38:16.624 INFO:teuthology.run_tasks:Running task internal.sudo... 2019-12-21T15:38:16.638 INFO:teuthology.task.internal:Configuring sudo... 2019-12-21T15:38:16.638 INFO:teuthology.orchestra.run.smithi172:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2019-12-21T15:38:16.692 INFO:teuthology.run_tasks:Running task internal.syslog... 2019-12-21T15:38:16.703 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2019-12-21T15:38:16.704 INFO:teuthology.orchestra.run.smithi172:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2019-12-21T15:38:16.746 INFO:teuthology.orchestra.run.smithi172:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2019-12-21T15:38:16.819 INFO:teuthology.orchestra.run.smithi172:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2019-12-21T15:38:16.878 INFO:teuthology.orchestra.run.smithi172:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2019-12-21T15:38:16.951 INFO:teuthology.orchestra.run.smithi172:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2019-12-21T15:38:17.000 INFO:teuthology.orchestra.run.smithi172:> sudo sh -c 'cat > /etc/rsyslog.d/80-cephtest.conf' 2019-12-21T15:38:17.086 INFO:teuthology.orchestra.run.smithi172:> sudo service rsyslog restart 2019-12-21T15:38:17.185 INFO:teuthology.orchestra.run.smithi172.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2019-12-21T15:38:17.537 INFO:teuthology.run_tasks:Running task internal.timer... 2019-12-21T15:38:17.549 INFO:teuthology.task.internal:Starting timer... 2019-12-21T15:38:17.550 INFO:teuthology.run_tasks:Running task pcp... 2019-12-21T15:38:17.600 INFO:teuthology.run_tasks:Running task selinux... 2019-12-21T15:38:17.614 INFO:teuthology.orchestra.run.smithi172:> sudo service auditd rotate 2019-12-21T15:38:17.669 INFO:teuthology.orchestra.run.smithi172.stdout:Rotating logs: [ OK ] 2019-12-21T15:38:17.671 DEBUG:teuthology.task.selinux:Getting current SELinux state 2019-12-21T15:38:17.671 INFO:teuthology.orchestra.run.smithi172:> /usr/sbin/getenforce 2019-12-21T15:38:17.693 INFO:teuthology.orchestra.run.smithi172.stdout:Permissive 2019-12-21T15:38:17.693 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi172.front.sepia.ceph.com': 'permissive'} 2019-12-21T15:38:17.693 INFO:teuthology.orchestra.run.smithi172:> sudo grep 'avc: .*denied' /var/log/audit/audit.log | grep -v '\(comm="dmidecode"\|chronyd.service\|name="cephtest"\|scontext=system_u:system_r:nrpe_t:s0\|scontext=system_u:system_r:pcp_pmlogger_t\|scontext=system_u:system_r:pcp_pmcd_t:s0\|comm="rhsmd"\|scontext=system_u:system_r:syslogd_t:s0\|tcontext=system_u:system_r:nrpe_t:s0\|comm="updatedb"\|comm="smartd"\)' 2019-12-21T15:38:17.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2019-12-21T15:38:17.761 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2019-12-21T15:38:17.761 INFO:teuthology.orchestra.run.smithi172:> sudo /usr/sbin/setenforce permissive 2019-12-21T15:38:17.829 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2019-12-21T15:38:17.854 DEBUG:teuthology.repo_utils:Setting repo remote to git://git.ceph.com/git/ceph-cm-ansible.git 2019-12-21T15:38:17.862 INFO:teuthology.repo_utils:Fetching master from origin 2019-12-21T15:38:17.927 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master to branch origin/master 2019-12-21T15:38:17.942 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}] 2019-12-21T15:38:17.945 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi172.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/cephlab.yml 2019-12-21T15:38:18.290 INFO:teuthology.task.ansible.out:Using /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/ansible.cfg as config file 2019-12-21T15:38:19.455 INFO:teuthology.task.ansible.out: [WARNING]: file /home/teuthworker/src/git.ceph.com_git_ceph-cm- ansible_master/roles/testnode/tasks/setup-opensuse.yml is empty and had no tasks to include 2019-12-21T15:38:22.031 INFO:teuthology.task.ansible.out: PLAY [all] ********************************************************************* 2019-12-21T15:38:22.053 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:38:22 +0000 (0:00:00.060) 0:00:00.060 ***** 2019-12-21T15:38:24.607 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* ok: [smithi172.front.sepia.ceph.com] 2019-12-21T15:38:24.703 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:38:24 +0000 (0:00:02.649) 0:00:02.710 ***** 2019-12-21T15:38:25.030 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the sudo group.] ******************************** 2019-12-21T15:38:25.031 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => {"changed": false, "gid": 1000, "name": "sudo", "state": "present", "system": false} 2019-12-21T15:38:25.073 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:38:25 +0000 (0:00:00.370) 0:00:03.080 ***** 2019-12-21T15:38:25.520 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the ansible user.] ****************************** 2019-12-21T15:38:25.521 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 1100, "groups": "sudo", "home": "/home/cm", "move_home": false, "name": "cm", "shell": "/bin/bash", "state": "present", "uid": 1100} 2019-12-21T15:38:25.557 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:38:25 +0000 (0:00:00.483) 0:00:03.565 ***** 2019-12-21T15:38:25.583 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Delete the ansible users password.] ******************** 2019-12-21T15:38:25.585 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-21T15:38:25.622 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:38:25 +0000 (0:00:00.064) 0:00:03.629 ***** 2019-12-21T15:38:25.965 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Ensure includedir is present in sudoers.] ************** ok: [smithi172.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2019-12-21T15:38:26.001 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:38:26 +0000 (0:00:00.379) 0:00:04.009 ***** 2019-12-21T15:38:26.933 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the cephlab_sudo sudoers.d file.] *************** 2019-12-21T15:38:26.934 INFO:teuthology.task.ansible.out:changed: [smithi172.front.sepia.ceph.com] => {"changed": true, "checksum": "6c8e8b7ecc895eb2094cafe75c34b58f183442a7", "dest": "/etc/sudoers.d/cephlab_sudo", "gid": 0, "group": "root", "md5sum": "c3657b9ef874d0ad278742c3210136a0", "mode": "0440", "owner": "root", "secontext": "system_u:object_r:etc_t:s0", "size": 182, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1576942706.02-179779686885401/source", "state": "file", "uid": 0} 2019-12-21T15:38:26.974 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:38:26 +0000 (0:00:00.972) 0:00:04.981 ***** 2019-12-21T15:38:27.512 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Add authorized keys for the ansible user.] ************* 2019-12-21T15:38:27.512 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item=ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmiHwkLkoJZiQs67KfE8jf0CdXjgbbpzKywUleMk4kWC7Wdn+92q/33DXpfS8xjg7opB9GTK3EW2C/RKBpf5RdaVG/5M7tRyUDLLbS3CURUoF6aNQI+8JIuFlsIfhO3Qw4uT7NlJCqsusrSbyLtGd/Im4CzwhvP1Enn6aUfyxv2yV4HoVjXbbIp2hlRXgcW8G/9q0TMuNfX4J6dpj9/PG8BbkKH/NReL2UMtjCGyRdIQvIwJRHMozjcE21824ZatA7nBeecX0MBRS6/A286uj6Z0PgmfJXYw0yw2Z2uSWR1ptIuMEiDBt96Pzn0dvfl7g/7bCi93MBQf/6sH9n4wkN root@typica001) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmiHwkLkoJZiQs67KfE8jf0CdXjgbbpzKywUleMk4kWC7Wdn+92q/33DXpfS8xjg7opB9GTK3EW2C/RKBpf5RdaVG/5M7tRyUDLLbS3CURUoF6aNQI+8JIuFlsIfhO3Qw4uT7NlJCqsusrSbyLtGd/Im4CzwhvP1Enn6aUfyxv2yV4HoVjXbbIp2hlRXgcW8G/9q0TMuNfX4J6dpj9/PG8BbkKH/NReL2UMtjCGyRdIQvIwJRHMozjcE21824ZatA7nBeecX0MBRS6/A286uj6Z0PgmfJXYw0yw2Z2uSWR1ptIuMEiDBt96Pzn0dvfl7g/7bCi93MBQf/6sH9n4wkN root@typica001", "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmiHwkLkoJZiQs67KfE8jf0CdXjgbbpzKywUleMk4kWC7Wdn+92q/33DXpfS8xjg7opB9GTK3EW2C/RKBpf5RdaVG/5M7tRyUDLLbS3CURUoF6aNQI+8JIuFlsIfhO3Qw4uT7NlJCqsusrSbyLtGd/Im4CzwhvP1Enn6aUfyxv2yV4HoVjXbbIp2hlRXgcW8G/9q0TMuNfX4J6dpj9/PG8BbkKH/NReL2UMtjCGyRdIQvIwJRHMozjcE21824ZatA7nBeecX0MBRS6/A286uj6Z0PgmfJXYw0yw2Z2uSWR1ptIuMEiDBt96Pzn0dvfl7g/7bCi93MBQf/6sH9n4wkN root@typica001", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2019-12-21T15:38:27.789 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item=ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com", "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2019-12-21T15:38:28.057 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item=ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local", "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2019-12-21T15:38:28.336 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item=ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus", "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2019-12-21T15:38:28.609 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item=ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541", "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2019-12-21T15:38:28.890 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item=ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50", "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2019-12-21T15:38:29.163 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item=ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2019-12-21T15:38:29.262 INFO:teuthology.task.ansible.out: PLAY [teuthology] ************************************************************** skipping: no hosts matched 2019-12-21T15:38:29.268 INFO:teuthology.task.ansible.out: 2019-12-21T15:38:29.268 INFO:teuthology.task.ansible.out:PLAY [testnodes] *************************************************************** 2019-12-21T15:38:29.479 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:38:29 +0000 (0:00:02.505) 0:00:07.486 ***** 2019-12-21T15:38:30.141 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* ok: [smithi172.front.sepia.ceph.com] 2019-12-21T15:38:30.224 INFO:teuthology.task.ansible.out: [WARNING]: Using run_once with the free strategy is not currently supported. This task will still be executed for every host in the inventory list. 2019-12-21T15:38:30.225 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:38:30 +0000 (0:00:00.744) 0:00:08.231 ***** 2019-12-21T15:38:30.732 INFO:teuthology.task.ansible.out: TASK [users : Check keys_repo HEAD sha1] *************************************** 2019-12-21T15:38:30.734 INFO:teuthology.task.ansible.out:changed: [smithi172.front.sepia.ceph.com -> localhost] => {"changed": true, "cmd": "git ls-remote https://github.com/ceph/keys HEAD | awk '{ print $1 }'", "delta": "0:00:00.224974", "end": "2019-12-21 15:38:30.709403", "rc": 0, "start": "2019-12-21 15:38:30.484429", "stderr": "", "stderr_lines": [], "stdout": "f4e10beea550ddcc5081090533acfa405cd36c43", "stdout_lines": ["f4e10beea550ddcc5081090533acfa405cd36c43"]} 2019-12-21T15:38:30.771 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:38:30 +0000 (0:00:00.547) 0:00:08.779 ***** 2019-12-21T15:38:30.974 INFO:teuthology.task.ansible.out: TASK [users : Check host's /keys-repo-sha1 sentinel file] ********************** 2019-12-21T15:38:30.975 INFO:teuthology.task.ansible.out:changed: [smithi172.front.sepia.ceph.com] => {"changed": true, "cmd": ["cat", "/keys-repo-sha1"], "delta": "0:00:00.010268", "end": "2019-12-21 15:38:30.954111", "failed_when_result": false, "rc": 0, "start": "2019-12-21 15:38:30.943843", "stderr": "", "stderr_lines": [], "stdout": "c84d080fc97e482f88d240859322d5765ac90366", "stdout_lines": ["c84d080fc97e482f88d240859322d5765ac90366"]} 2019-12-21T15:38:31.013 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:38:31 +0000 (0:00:00.241) 0:00:09.020 ***** 2019-12-21T15:38:31.039 INFO:teuthology.task.ansible.out: TASK [users : Determine if we can skip users and pubkeys updates] ************** 2019-12-21T15:38:31.041 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-21T15:38:31.079 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:38:31 +0000 (0:00:00.065) 0:00:09.086 ***** 2019-12-21T15:38:31.105 INFO:teuthology.task.ansible.out: TASK [users : Merge extra_admin_users into managed_admin_users] **************** 2019-12-21T15:38:31.106 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-21T15:38:31.144 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:38:31 +0000 (0:00:00.065) 0:00:09.152 ***** 2019-12-21T15:38:31.170 INFO:teuthology.task.ansible.out: TASK [users : Remove managed_admin_users from managed_users] ******************* 2019-12-21T15:38:31.172 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-21T15:38:31.210 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:38:31 +0000 (0:00:00.065) 0:00:09.217 ***** 2019-12-21T15:38:31.237 INFO:teuthology.task.ansible.out: TASK [users : Filter the managed_users list] *********************************** 2019-12-21T15:38:31.238 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-21T15:38:31.283 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:38:31 +0000 (0:00:00.073) 0:00:09.290 ***** 2019-12-21T15:38:31.309 INFO:teuthology.task.ansible.out: TASK [users : Filter the managed_admin_users list] ***************************** 2019-12-21T15:38:31.311 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-21T15:38:31.348 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:38:31 +0000 (0:00:00.064) 0:00:09.355 ***** 2019-12-21T15:38:31.550 INFO:teuthology.task.ansible.out: TASK [users : Set UID_MIN to 1001] ********************************************* 2019-12-21T15:38:31.551 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2019-12-21T15:38:31.599 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:38:31 +0000 (0:00:00.250) 0:00:09.606 ***** 2019-12-21T15:38:31.839 INFO:teuthology.task.ansible.out: TASK [users : Create all admin users with sudo access.] ************************ 2019-12-21T15:38:31.840 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9', u'name': u'andrewschoen'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1101, "groups": "sudo", "home": "/home/andrewschoen", "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "move_home": false, "name": "andrewschoen", "shell": "/bin/bash", "state": "present", "uid": 1101} 2019-12-21T15:38:32.036 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n', u'name': u'zack'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1102, "groups": "sudo", "home": "/home/zack", "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "move_home": false, "name": "zack", "shell": "/bin/bash", "state": "present", "uid": 1102} 2019-12-21T15:38:32.236 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd', u'name': u'kdreyer'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1103, "groups": "sudo", "home": "/home/kdreyer", "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "move_home": false, "name": "kdreyer", "shell": "/bin/bash", "state": "present", "uid": 1103} 2019-12-21T15:38:32.434 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'alfredo@papaya.local V5VkRT2Du88WwQrYeK196w 4dff4d9bdc4d9ab60532814c338abe4534cb45a9ca729bdbc2411ad513516b47', u'name': u'adeza'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1104, "groups": "sudo", "home": "/home/adeza", "item": {"name": "adeza", "ovpn": "alfredo@papaya.local V5VkRT2Du88WwQrYeK196w 4dff4d9bdc4d9ab60532814c338abe4534cb45a9ca729bdbc2411ad513516b47"}, "move_home": false, "name": "adeza", "shell": "/bin/bash", "state": "present", "uid": 1104} 2019-12-21T15:38:32.635 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\n', u'name': u'dmick'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1105, "groups": "sudo", "home": "/home/dmick", "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\n"}, "move_home": false, "name": "dmick", "shell": "/bin/bash", "state": "present", "uid": 1105} 2019-12-21T15:38:32.836 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n', u'name': u'sage'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1106, "groups": "sudo", "home": "/home/sage", "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "move_home": false, "name": "sage", "shell": "/bin/bash", "state": "present", "uid": 1106} 2019-12-21T15:38:33.044 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n', u'name': u'dgalloway'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1107, "groups": "sudo", "home": "/home/dgalloway", "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "move_home": false, "name": "dgalloway", "shell": "/bin/bash", "state": "present", "uid": 1107} 2019-12-21T15:38:33.246 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045', u'name': u'akraitma'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1108, "groups": "sudo", "home": "/home/akraitma", "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "move_home": false, "name": "akraitma", "shell": "/bin/bash", "state": "present", "uid": 1108} 2019-12-21T15:38:33.455 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e', u'name': u'trociny'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1109, "groups": "sudo", "home": "/home/trociny", "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "move_home": false, "name": "trociny", "shell": "/bin/bash", "state": "present", "uid": 1109} 2019-12-21T15:38:33.651 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007', u'name': u'smithfarm', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1110, "groups": "sudo", "home": "/home/smithfarm", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "move_home": false, "name": "smithfarm", "shell": "/bin/bash", "state": "present", "uid": 1110} 2019-12-21T15:38:33.848 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n', u'name': u'mbenjamin'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1111, "groups": "sudo", "home": "/home/mbenjamin", "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "move_home": false, "name": "mbenjamin", "shell": "/bin/bash", "state": "present", "uid": 1111} 2019-12-21T15:38:34.045 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01', u'name': u'aemerson', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1112, "groups": "sudo", "home": "/home/aemerson", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt", "name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01"}, "move_home": false, "name": "aemerson", "shell": "/bin/bash", "state": "present", "uid": 1112} 2019-12-21T15:38:34.246 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78', u'name': u'amaredia', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1113, "groups": "sudo", "home": "/home/amaredia", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "move_home": false, "name": "amaredia", "shell": "/bin/bash", "state": "present", "uid": 1113} 2019-12-21T15:38:34.454 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5', u'name': u'tserlin', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1114, "groups": "sudo", "home": "/home/tserlin", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "move_home": false, "name": "tserlin", "shell": "/bin/bash", "state": "present", "uid": 1114} 2019-12-21T15:38:34.662 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28', u'name': u'dis'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1115, "groups": "sudo", "home": "/home/dis", "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "move_home": false, "name": "dis", "shell": "/bin/bash", "state": "present", "uid": 1115} 2019-12-21T15:38:34.879 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n', u'name': u'gregf'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1116, "groups": "sudo", "home": "/home/gregf", "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "move_home": false, "name": "gregf", "shell": "/bin/bash", "state": "present", "uid": 1116} 2019-12-21T15:38:35.071 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n', u'name': u'joshd'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1117, "groups": "sudo", "home": "/home/joshd", "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "move_home": false, "name": "joshd", "shell": "/bin/bash", "state": "present", "uid": 1117} 2019-12-21T15:38:35.277 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n', u'name': u'davidz'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1118, "groups": "sudo", "home": "/home/davidz", "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "move_home": false, "name": "davidz", "shell": "/bin/bash", "state": "present", "uid": 1118} 2019-12-21T15:38:35.474 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'gmeno@gmeno-virtual-machine FKFu8B2pMqotpmEVAO1few 8229574e499eaf767a408909f5afdf2e2a0bb8f3e61b18d63a651f7102c68dbc', u'name': u'gmeno'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1119, "groups": "sudo", "home": "/home/gmeno", "item": {"name": "gmeno", "ovpn": "gmeno@gmeno-virtual-machine FKFu8B2pMqotpmEVAO1few 8229574e499eaf767a408909f5afdf2e2a0bb8f3e61b18d63a651f7102c68dbc"}, "move_home": false, "name": "gmeno", "shell": "/bin/bash", "state": "present", "uid": 1119} 2019-12-21T15:38:35.681 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1', u'name': u'ivancich'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1120, "groups": "sudo", "home": "/home/ivancich", "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "move_home": false, "name": "ivancich", "shell": "/bin/bash", "state": "present", "uid": 1120} 2019-12-21T15:38:35.877 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n', u'name': u'wusui'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1121, "groups": "sudo", "home": "/home/wusui", "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "move_home": false, "name": "wusui", "shell": "/bin/bash", "state": "present", "uid": 1121} 2019-12-21T15:38:36.084 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304', u'name': u'zyan'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1122, "groups": "sudo", "home": "/home/zyan", "item": {"name": "zyan", "ovpn": "zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304"}, "move_home": false, "name": "zyan", "shell": "/bin/bash", "state": "present", "uid": 1122} 2019-12-21T15:38:36.281 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH 5ivdxgFO4eIkbXVhl8xkvw 59212d29b8b42d9fe457c1b2c43d774e1d25807be10dcc1252d4aec63b97a467\n', u'name': u'yuriw'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1123, "groups": "sudo", "home": "/home/yuriw", "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH 5ivdxgFO4eIkbXVhl8xkvw 59212d29b8b42d9fe457c1b2c43d774e1d25807be10dcc1252d4aec63b97a467\n"}, "move_home": false, "name": "yuriw", "shell": "/bin/bash", "state": "present", "uid": 1123} 2019-12-21T15:38:36.477 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n', u'name': u'tamil'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1124, "groups": "sudo", "home": "/home/tamil", "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "move_home": false, "name": "tamil", "shell": "/bin/bash", "state": "present", "uid": 1124} 2019-12-21T15:38:36.694 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n', u'name': u'jowilkin'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1125, "groups": "sudo", "home": "/home/jowilkin", "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "move_home": false, "name": "jowilkin", "shell": "/bin/bash", "state": "present", "uid": 1125} 2019-12-21T15:38:36.890 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4', u'name': u'bhubbard'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1126, "groups": "sudo", "home": "/home/bhubbard", "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "move_home": false, "name": "bhubbard", "shell": "/bin/bash", "state": "present", "uid": 1126} 2019-12-21T15:38:37.096 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n', u'name': u'yehudasa'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1127, "groups": "sudo", "home": "/home/yehudasa", "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "move_home": false, "name": "yehudasa", "shell": "/bin/bash", "state": "present", "uid": 1127} 2019-12-21T15:38:37.293 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4', u'name': u'dang'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1128, "groups": "sudo", "home": "/home/dang", "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "move_home": false, "name": "dang", "shell": "/bin/bash", "state": "present", "uid": 1128} 2019-12-21T15:38:37.504 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb', u'name': u'branto'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1129, "groups": "sudo", "home": "/home/branto", "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "move_home": false, "name": "branto", "shell": "/bin/bash", "state": "present", "uid": 1129} 2019-12-21T15:38:37.718 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6', u'name': u'xiaoxichen'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1130, "groups": "sudo", "home": "/home/xiaoxichen", "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "move_home": false, "name": "xiaoxichen", "shell": "/bin/bash", "state": "present", "uid": 1130} 2019-12-21T15:38:37.916 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e', u'name': u'ffilz'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1131, "groups": "sudo", "home": "/home/ffilz", "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "move_home": false, "name": "ffilz", "shell": "/bin/bash", "state": "present", "uid": 1131} 2019-12-21T15:38:38.124 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n', u'name': u'joao'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1132, "groups": "sudo", "home": "/home/joao", "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "move_home": false, "name": "joao", "shell": "/bin/bash", "state": "present", "uid": 1132} 2019-12-21T15:38:38.323 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n', u'name': u'nhm'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1133, "groups": "sudo", "home": "/home/nhm", "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "move_home": false, "name": "nhm", "shell": "/bin/bash", "state": "present", "uid": 1133} 2019-12-21T15:38:38.520 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313', u'name': u'nwatkins'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1134, "groups": "sudo", "home": "/home/nwatkins", "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "move_home": false, "name": "nwatkins", "shell": "/bin/bash", "state": "present", "uid": 1134} 2019-12-21T15:38:38.727 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22', u'name': u'mkidd'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1135, "groups": "sudo", "home": "/home/mkidd", "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "move_home": false, "name": "mkidd", "shell": "/bin/bash", "state": "present", "uid": 1135} 2019-12-21T15:38:38.927 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af', u'name': u'jlopez'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1136, "groups": "sudo", "home": "/home/jlopez", "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "move_home": false, "name": "jlopez", "shell": "/bin/bash", "state": "present", "uid": 1136} 2019-12-21T15:38:39.136 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb', u'name': u'haomaiwang'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1137, "groups": "sudo", "home": "/home/haomaiwang", "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "move_home": false, "name": "haomaiwang", "shell": "/bin/bash", "state": "present", "uid": 1137} 2019-12-21T15:38:39.344 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608', u'name': u'jdillaman'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1138, "groups": "sudo", "home": "/home/jdillaman", "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "move_home": false, "name": "jdillaman", "shell": "/bin/bash", "state": "present", "uid": 1138} 2019-12-21T15:38:39.553 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3', u'name': u'kchai'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1139, "groups": "sudo", "home": "/home/kchai", "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "move_home": false, "name": "kchai", "shell": "/bin/bash", "state": "present", "uid": 1139} 2019-12-21T15:38:39.768 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2', u'name': u'vumrao'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1140, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1140} 2019-12-21T15:38:39.964 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n', u'name': u'dfuller'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1141, "groups": "sudo", "home": "/home/dfuller", "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "move_home": false, "name": "dfuller", "shell": "/bin/bash", "state": "present", "uid": 1141} 2019-12-21T15:38:40.173 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n', u'name': u'owasserm'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1142, "groups": "sudo", "home": "/home/owasserm", "item": {"name": "owasserm", "ovpn": "owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n"}, "move_home": false, "name": "owasserm", "shell": "/bin/bash", "state": "present", "uid": 1142} 2019-12-21T15:38:40.370 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n', u'name': u'vasu'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1143, "groups": "sudo", "home": "/home/vasu", "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "move_home": false, "name": "vasu", "shell": "/bin/bash", "state": "present", "uid": 1143} 2019-12-21T15:38:40.581 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37', u'name': u'smohan'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1144, "groups": "sudo", "home": "/home/smohan", "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "move_home": false, "name": "smohan", "shell": "/bin/bash", "state": "present", "uid": 1144} 2019-12-21T15:38:40.789 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114', u'name': u'abhi'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1145, "groups": "sudo", "home": "/home/abhi", "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "move_home": false, "name": "abhi", "shell": "/bin/bash", "state": "present", "uid": 1145} 2019-12-21T15:38:40.996 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611', u'name': u'cbodley'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1146, "groups": "sudo", "home": "/home/cbodley", "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "move_home": false, "name": "cbodley", "shell": "/bin/bash", "state": "present", "uid": 1146} 2019-12-21T15:38:41.195 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195', u'name': u'mwatts'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1147, "groups": "sudo", "home": "/home/mwatts", "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "move_home": false, "name": "mwatts", "shell": "/bin/bash", "state": "present", "uid": 1147} 2019-12-21T15:38:41.403 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6', u'name': u'prsrivas'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1148, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1148} 2019-12-21T15:38:41.608 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea', u'name': u'pdonnell'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1149, "groups": "sudo", "home": "/home/pdonnell", "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "move_home": false, "name": "pdonnell", "shell": "/bin/bash", "state": "present", "uid": 1149} 2019-12-21T15:38:41.810 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868', u'name': u'jlayton'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1150, "groups": "sudo", "home": "/home/jlayton", "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "move_home": false, "name": "jlayton", "shell": "/bin/bash", "state": "present", "uid": 1150} 2019-12-21T15:38:42.008 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f', u'name': u'rzarzynski'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1151, "groups": "sudo", "home": "/home/rzarzynski", "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "move_home": false, "name": "rzarzynski", "shell": "/bin/bash", "state": "present", "uid": 1151} 2019-12-21T15:38:42.206 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e', u'name': u'rdias'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1152, "groups": "sudo", "home": "/home/rdias", "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "move_home": false, "name": "rdias", "shell": "/bin/bash", "state": "present", "uid": 1152} 2019-12-21T15:38:42.406 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953', u'name': u'vshankar'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1153, "groups": "sudo", "home": "/home/vshankar", "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953"}, "move_home": false, "name": "vshankar", "shell": "/bin/bash", "state": "present", "uid": 1153} 2019-12-21T15:38:42.618 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58', u'name': u'akupczyk'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1154, "groups": "sudo", "home": "/home/akupczyk", "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "move_home": false, "name": "akupczyk", "shell": "/bin/bash", "state": "present", "uid": 1154} 2019-12-21T15:38:42.821 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e', u'name': u'nojha'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1155, "groups": "sudo", "home": "/home/nojha", "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e"}, "move_home": false, "name": "nojha", "shell": "/bin/bash", "state": "present", "uid": 1155} 2019-12-21T15:38:43.032 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43', u'name': u'ifed01'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1156, "groups": "sudo", "home": "/home/ifed01", "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "move_home": false, "name": "ifed01", "shell": "/bin/bash", "state": "present", "uid": 1156} 2019-12-21T15:38:43.231 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4', u'name': u'myoungwon'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1157, "groups": "sudo", "home": "/home/myoungwon", "item": {"name": "myoungwon", "ovpn": "myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4"}, "move_home": false, "name": "myoungwon", "shell": "/bin/bash", "state": "present", "uid": 1157} 2019-12-21T15:38:43.441 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2', u'name': u'gabrioux'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1158, "groups": "sudo", "home": "/home/gabrioux", "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "move_home": false, "name": "gabrioux", "shell": "/bin/bash", "state": "present", "uid": 1158} 2019-12-21T15:38:43.654 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f', u'name': u'leseb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1159, "groups": "sudo", "home": "/home/leseb", "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "move_home": false, "name": "leseb", "shell": "/bin/bash", "state": "present", "uid": 1159} 2019-12-21T15:38:43.856 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca', u'name': u'hchen'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1160, "groups": "sudo", "home": "/home/hchen", "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "move_home": false, "name": "hchen", "shell": "/bin/bash", "state": "present", "uid": 1160} 2019-12-21T15:38:44.056 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n', u'name': u'jcollin'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1161, "groups": "sudo", "home": "/home/jcollin", "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "move_home": false, "name": "jcollin", "shell": "/bin/bash", "state": "present", "uid": 1161} 2019-12-21T15:38:44.255 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc', u'name': u'xxg'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1162, "groups": "sudo", "home": "/home/xxg", "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "move_home": false, "name": "xxg", "shell": "/bin/bash", "state": "present", "uid": 1162} 2019-12-21T15:38:44.463 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53', u'name': u'pcuzner'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1163, "groups": "sudo", "home": "/home/pcuzner", "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "move_home": false, "name": "pcuzner", "shell": "/bin/bash", "state": "present", "uid": 1163} 2019-12-21T15:38:44.664 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2', u'name': u'mkogan'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1164, "groups": "sudo", "home": "/home/mkogan", "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "move_home": false, "name": "mkogan", "shell": "/bin/bash", "state": "present", "uid": 1164} 2019-12-21T15:38:44.870 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed', u'name': u'vumrao'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1140, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1140} 2019-12-21T15:38:45.066 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736', u'name': u'henrix'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1165, "groups": "sudo", "home": "/home/henrix", "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "move_home": false, "name": "henrix", "shell": "/bin/bash", "state": "present", "uid": 1165} 2019-12-21T15:38:45.262 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26', u'name': u'pbs1108'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1166, "groups": "sudo", "home": "/home/pbs1108", "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "move_home": false, "name": "pbs1108", "shell": "/bin/bash", "state": "present", "uid": 1166} 2019-12-21T15:38:45.473 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315', u'name': u'clacroix'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1167, "groups": "sudo", "home": "/home/clacroix", "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "move_home": false, "name": "clacroix", "shell": "/bin/bash", "state": "present", "uid": 1167} 2019-12-21T15:38:45.682 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006', u'name': u'epuertat'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1168, "groups": "sudo", "home": "/home/epuertat", "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "move_home": false, "name": "epuertat", "shell": "/bin/bash", "state": "present", "uid": 1168} 2019-12-21T15:38:45.889 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461', u'name': u'tdehler'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1169, "groups": "sudo", "home": "/home/tdehler", "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "move_home": false, "name": "tdehler", "shell": "/bin/bash", "state": "present", "uid": 1169} 2019-12-21T15:38:46.087 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305', u'name': u'laura'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1170, "groups": "sudo", "home": "/home/laura", "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "move_home": false, "name": "laura", "shell": "/bin/bash", "state": "present", "uid": 1170} 2019-12-21T15:38:46.285 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce', u'name': u'yaarit'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1171, "groups": "sudo", "home": "/home/yaarit", "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "move_home": false, "name": "yaarit", "shell": "/bin/bash", "state": "present", "uid": 1171} 2019-12-21T15:38:46.484 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3', u'name': u'rpavani1998'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1172, "groups": "sudo", "home": "/home/rpavani1998", "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "move_home": false, "name": "rpavani1998", "shell": "/bin/bash", "state": "present", "uid": 1172} 2019-12-21T15:38:46.684 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4', u'name': u'rishabh'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1173, "groups": "sudo", "home": "/home/rishabh", "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "move_home": false, "name": "rishabh", "shell": "/bin/bash", "state": "present", "uid": 1173} 2019-12-21T15:38:46.902 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2', u'name': u'skrah'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1174, "groups": "sudo", "home": "/home/skrah", "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "move_home": false, "name": "skrah", "shell": "/bin/bash", "state": "present", "uid": 1174} 2019-12-21T15:38:47.110 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8', u'name': u'smanjara'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1175, "groups": "sudo", "home": "/home/smanjara", "item": {"name": "smanjara", "ovpn": "smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8"}, "move_home": false, "name": "smanjara", "shell": "/bin/bash", "state": "present", "uid": 1175} 2019-12-21T15:38:47.310 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41', u'name': u'bengland'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1176, "groups": "sudo", "home": "/home/bengland", "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "move_home": false, "name": "bengland", "shell": "/bin/bash", "state": "present", "uid": 1176} 2019-12-21T15:38:47.512 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d', u'name': u'pnawracay'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1177, "groups": "sudo", "home": "/home/pnawracay", "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "move_home": false, "name": "pnawracay", "shell": "/bin/bash", "state": "present", "uid": 1177} 2019-12-21T15:38:47.713 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8', u'name': u'alfonsomthd'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1178, "groups": "sudo", "home": "/home/alfonsomthd", "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8"}, "move_home": false, "name": "alfonsomthd", "shell": "/bin/bash", "state": "present", "uid": 1178} 2019-12-21T15:38:47.911 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e', u'name': u'oliveiradan'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1179, "groups": "sudo", "home": "/home/oliveiradan", "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "move_home": false, "name": "oliveiradan", "shell": "/bin/bash", "state": "present", "uid": 1179} 2019-12-21T15:38:48.109 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba', u'name': u'swagner'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1180, "groups": "sudo", "home": "/home/swagner", "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "move_home": false, "name": "swagner", "shell": "/bin/bash", "state": "present", "uid": 1180} 2019-12-21T15:38:48.306 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e', u'name': u'yuvalif'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1181, "groups": "sudo", "home": "/home/yuvalif", "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e"}, "move_home": false, "name": "yuvalif", "shell": "/bin/bash", "state": "present", "uid": 1181} 2019-12-21T15:38:48.515 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852', u'name': u'kkeithle'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1182, "groups": "sudo", "home": "/home/kkeithle", "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "move_home": false, "name": "kkeithle", "shell": "/bin/bash", "state": "present", "uid": 1182} 2019-12-21T15:38:48.719 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130', u'name': u'emmericp'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1183, "groups": "sudo", "home": "/home/emmericp", "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "move_home": false, "name": "emmericp", "shell": "/bin/bash", "state": "present", "uid": 1183} 2019-12-21T15:38:48.926 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0', u'name': u'mchangir'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1184, "groups": "sudo", "home": "/home/mchangir", "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "move_home": false, "name": "mchangir", "shell": "/bin/bash", "state": "present", "uid": 1184} 2019-12-21T15:38:49.124 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286', u'name': u'sidharthanup'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1185, "groups": "sudo", "home": "/home/sidharthanup", "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "move_home": false, "name": "sidharthanup", "shell": "/bin/bash", "state": "present", "uid": 1185} 2019-12-21T15:38:49.332 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31', u'name': u'varsha'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1186, "groups": "sudo", "home": "/home/varsha", "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "move_home": false, "name": "varsha", "shell": "/bin/bash", "state": "present", "uid": 1186} 2019-12-21T15:38:49.531 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n', u'name': u'sjust'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1187, "groups": "sudo", "home": "/home/sjust", "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "move_home": false, "name": "sjust", "shell": "/bin/bash", "state": "present", "uid": 1187} 2019-12-21T15:38:49.738 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'deepika@thinkpad HXRUSerxILmV0ejAD/Zt1g 29b44e741567e7a2e7b664d6533247d79851fc302051c9ba34103c63a2fcfacc', u'name': u'ideepika'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1188, "groups": "sudo", "home": "/home/ideepika", "item": {"name": "ideepika", "ovpn": "deepika@thinkpad HXRUSerxILmV0ejAD/Zt1g 29b44e741567e7a2e7b664d6533247d79851fc302051c9ba34103c63a2fcfacc"}, "move_home": false, "name": "ideepika", "shell": "/bin/bash", "state": "present", "uid": 1188} 2019-12-21T15:38:49.934 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46', u'name': u'gsalomon'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1189, "groups": "sudo", "home": "/home/gsalomon", "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "move_home": false, "name": "gsalomon", "shell": "/bin/bash", "state": "present", "uid": 1189} 2019-12-21T15:38:50.131 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de', u'name': u'soumyakoduri'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1190, "groups": "sudo", "home": "/home/soumyakoduri", "item": {"name": "soumyakoduri", "ovpn": "skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de"}, "move_home": false, "name": "soumyakoduri", "shell": "/bin/bash", "state": "present", "uid": 1190} 2019-12-21T15:38:50.329 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a', u'name': u'kyr'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1191, "groups": "sudo", "home": "/home/kyr", "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "move_home": false, "name": "kyr", "shell": "/bin/bash", "state": "present", "uid": 1191} 2019-12-21T15:38:50.529 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849', u'name': u'sseshasa'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1192, "groups": "sudo", "home": "/home/sseshasa", "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "move_home": false, "name": "sseshasa", "shell": "/bin/bash", "state": "present", "uid": 1192} 2019-12-21T15:38:50.736 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189', u'name': u'rfriedma'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1193, "groups": "sudo", "home": "/home/rfriedma", "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "move_home": false, "name": "rfriedma", "shell": "/bin/bash", "state": "present", "uid": 1193} 2019-12-21T15:38:50.926 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b', u'name': u'lmb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1194, "groups": "sudo", "home": "/home/lmb", "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "move_home": false, "name": "lmb", "shell": "/bin/bash", "state": "present", "uid": 1194} 2019-12-21T15:38:51.132 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc', u'name': u'ksirivad'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1195, "groups": "sudo", "home": "/home/ksirivad", "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc"}, "move_home": false, "name": "ksirivad", "shell": "/bin/bash", "state": "present", "uid": 1195} 2019-12-21T15:38:51.333 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4', u'name': u'ofriedma'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1196, "groups": "sudo", "home": "/home/ofriedma", "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "move_home": false, "name": "ofriedma", "shell": "/bin/bash", "state": "present", "uid": 1196} 2019-12-21T15:38:51.539 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52', u'name': u'h4iresis'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1197, "groups": "sudo", "home": "/home/h4iresis", "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "move_home": false, "name": "h4iresis", "shell": "/bin/bash", "state": "present", "uid": 1197} 2019-12-21T15:38:51.735 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5', u'name': u'shyukri'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1198, "groups": "sudo", "home": "/home/shyukri", "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "move_home": false, "name": "shyukri", "shell": "/bin/bash", "state": "present", "uid": 1198} 2019-12-21T15:38:51.933 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e', u'name': u'scarvalhojr'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1199, "groups": "sudo", "home": "/home/scarvalhojr", "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "move_home": false, "name": "scarvalhojr", "shell": "/bin/bash", "state": "present", "uid": 1199} 2019-12-21T15:38:52.129 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b', u'name': u'rakesh'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1200, "groups": "sudo", "home": "/home/rakesh", "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "move_home": false, "name": "rakesh", "shell": "/bin/bash", "state": "present", "uid": 1200} 2019-12-21T15:38:52.327 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f', u'name': u'ryg-'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1201, "groups": "sudo", "home": "/home/ryg-", "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "move_home": false, "name": "ryg-", "shell": "/bin/bash", "state": "present", "uid": 1201} 2019-12-21T15:38:52.524 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3', u'name': u'prsrivas'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1148, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1148} 2019-12-21T15:38:52.730 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'zdover@slim_thinkpad H/78rBqDIJ5AAjMT6sTmKg 49fb166c553d14549491280458269f987bea97295dbf525ffb7c35336175ee42', u'name': u'zdover'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1202, "groups": "sudo", "home": "/home/zdover", "item": {"name": "zdover", "ovpn": "zdover@slim_thinkpad H/78rBqDIJ5AAjMT6sTmKg 49fb166c553d14549491280458269f987bea97295dbf525ffb7c35336175ee42"}, "move_home": false, "name": "zdover", "shell": "/bin/bash", "state": "present", "uid": 1202} 2019-12-21T15:38:52.928 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'rraja@thinkpadp50 8S3Lez75woSBdYiLqQxf2g c9d1666dfc7a3b762700cd250f775a506c397b0f0e6a86289ba555a3546e532b', u'name': u'rraja'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1203, "groups": "sudo", "home": "/home/rraja", "item": {"name": "rraja", "ovpn": "rraja@thinkpadp50 8S3Lez75woSBdYiLqQxf2g c9d1666dfc7a3b762700cd250f775a506c397b0f0e6a86289ba555a3546e532b"}, "move_home": false, "name": "rraja", "shell": "/bin/bash", "state": "present", "uid": 1203} 2019-12-21T15:38:53.126 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f', u'name': u'toabctl'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1204, "groups": "sudo", "home": "/home/toabctl", "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "move_home": false, "name": "toabctl", "shell": "/bin/bash", "state": "present", "uid": 1204} 2019-12-21T15:38:54.535 INFO:teuthology.task.ansible.out:changed: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'mgfritch@laptop 3KhqWETFJaVsH/I8FOBfUg 0aca6c3c6d45c554bad8ba9853f0bc575c9915eee33ee6be2059568ad09a9d72', u'name': u'mgfritch'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10102, "groups": "sudo", "home": "/home/mgfritch", "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 3KhqWETFJaVsH/I8FOBfUg 0aca6c3c6d45c554bad8ba9853f0bc575c9915eee33ee6be2059568ad09a9d72"}, "name": "mgfritch", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10102} 2019-12-21T15:38:54.708 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:38:54 +0000 (0:00:23.109) 0:00:32.716 ***** 2019-12-21T15:38:54.736 INFO:teuthology.task.ansible.out: TASK [users : Create all users without sudo access.] *************************** 2019-12-21T15:38:54.772 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:38:54 +0000 (0:00:00.064) 0:00:32.780 ***** 2019-12-21T15:38:54.820 INFO:teuthology.task.ansible.out: TASK [users : Merge managed_users and managed_admin users] ********************* 2019-12-21T15:38:54.824 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => {"ansible_facts": {"pubkey_users": [{"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, {"name": "adeza", "ovpn": "alfredo@papaya.local V5VkRT2Du88WwQrYeK196w 4dff4d9bdc4d9ab60532814c338abe4534cb45a9ca729bdbc2411ad513516b47"}, {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\n"}, {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, {"name": "tr 2019-12-21T15:38:54.825 INFO:teuthology.task.ansible.out:ociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt", "name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7 2019-12-21T15:38:54.826 INFO:teuthology.task.ansible.out:L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32 2019-12-21T15:38:54.830 INFO:teuthology.task.ansible.out:f606a33fb488ff621e11aa305ac979501030202658\n"}, {"name": "gmeno", "ovpn": "gmeno@gmeno-virtual-machine FKFu8B2pMqotpmEVAO1few 8229574e499eaf767a408909f5afdf2e2a0bb8f3e61b18d63a651f7102c68dbc"}, {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, {"name": "zyan", "ovpn": "zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304"}, {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH 5ivdxgFO4eIkbXVhl8xkvw 59212d29b8b42d9fe457c1b2c43d774e1d25807be10dcc1252d4aec63b97a467\n"}, {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b 2019-12-21T15:38:54.831 INFO:teuthology.task.ansible.out:09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e 2019-12-21T15:38:54.833 INFO:teuthology.task.ansible.out:66b842a921169f20a00481a4cd3d9cb"}, {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, {"name": "owasserm", "ovpn": "owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n"}, {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa 2019-12-21T15:38:54.834 INFO:teuthology.task.ansible.out:8ea"}, {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953"}, {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e"}, {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, {"name": "myoungwon", "ovpn": "myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4"}, {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 2019-12-21T15:38:54.834 INFO:teuthology.task.ansible.out:6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f 2019-12-21T15:38:54.836 INFO:teuthology.task.ansible.out:826aae1caa53"}, {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, {"name": "smanjara", "ovpn": "smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8"}, {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd1 2019-12-21T15:38:54.837 INFO:teuthology.task.ansible.out:7f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8"}, {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e"}, {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, {"name": "ideepika", "ovpn": "deepika@thinkpad HXRUSerxILmV0ejAD/Zt1g 29b44e741567e7a2e7b664d6533247d79851fc302051c9ba34103c63a2fcfacc"}, {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 2019-12-21T15:38:54.838 INFO:teuthology.task.ansible.out: 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, {"name": "soumyakoduri", "ovpn": "skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de"}, {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc"}, {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, {"name": "zdover", "ovpn": "zdover@slim_thinkpad H/78rBqDIJ5AAjMT6sTmKg 49fb166c553d14549491280458269f987bea97295dbf525ffb7c35336175ee42"}, {"name": "rraja", "ovpn" 2019-12-21T15:38:54.840 INFO:teuthology.task.ansible.out:: "rraja@thinkpadp50 8S3Lez75woSBdYiLqQxf2g c9d1666dfc7a3b762700cd250f775a506c397b0f0e6a86289ba555a3546e532b"}, {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, {"name": "mgfritch", "ovpn": "mgfritch@laptop 3KhqWETFJaVsH/I8FOBfUg 0aca6c3c6d45c554bad8ba9853f0bc575c9915eee33ee6be2059568ad09a9d72"}]}, "changed": false} 2019-12-21T15:38:54.878 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:38:54 +0000 (0:00:00.105) 0:00:32.886 ***** 2019-12-21T15:38:55.895 INFO:teuthology.task.ansible.out: TASK [users : Clone the keys repo] ********************************************* 2019-12-21T15:38:55.896 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com -> localhost] => {"after": "f4e10beea550ddcc5081090533acfa405cd36c43", "attempts": 1, "before": "f4e10beea550ddcc5081090533acfa405cd36c43", "changed": false, "remote_url_changed": false} 2019-12-21T15:38:55.939 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:38:55 +0000 (0:00:01.060) 0:00:33.946 ***** 2019-12-21T15:38:56.295 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2019-12-21T15:38:56.296 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9', u'name': u'andrewschoen'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com", "key_options": null, "keyfile": "/home/andrewschoen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "andrewschoen", "validate_certs": true} 2019-12-21T15:38:56.607 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n', u'name': u'zack'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local", "key_options": null, "keyfile": "/home/zack/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zack", "validate_certs": true} 2019-12-21T15:38:56.887 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd', u'name': u'kdreyer'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4eHmz10szeHNS3dNejKokW85ksB+iR4HGOFsmQM11Ni68Nm5aqEKvkOZU8TpY92vpCQL0A68GlrXB845cACdyk6HUJYyNNNMC43l1FYWOwjMqQBSdj8W3VQDTA6eiG60mt5fgI8fyR38rKzIA1MnTBkSSjuh5kQVJ9bdEp3GuY5oc8vxDNBlGJ6LYnyEWt/pqL2J+mpjqnOjsC+EbE2exhP9O+mvzpQiyo/+dEN1COwX3//pNRXGfOSeOczHNsJE8Eu+j/n/BlW57++sJyFMkzS7bUxMSGM6quvjQZ7RT1c5JM6vLEiQyzQxoRgzY93h1yKlOstBi0NamtpqHQZGP kdreyer@mbp.ktdreyer.com", "key_options": null, "keyfile": "/home/kdreyer/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kdreyer", "validate_certs": true} 2019-12-21T15:38:57.166 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'alfredo@papaya.local V5VkRT2Du88WwQrYeK196w 4dff4d9bdc4d9ab60532814c338abe4534cb45a9ca729bdbc2411ad513516b47', u'name': u'adeza'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "adeza", "ovpn": "alfredo@papaya.local V5VkRT2Du88WwQrYeK196w 4dff4d9bdc4d9ab60532814c338abe4534cb45a9ca729bdbc2411ad513516b47"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDCWy2i6Iv6IWLh53Z44O4fQ0ctxR8odCAWIZAaSQXAxxC++RS+95Zr+BZ5wJwsBOyP1G4AAhl+9uwnn3NmDDG7fKGexPoQrTZZ27KdJ3bHpZRq5LJf9byWFwz4rDsqIqkGjI9sUkfa6nV7ci2BdEkIgLupTT2uXUzTb/njOjI7Fc+cQ7ac4QfJ+yeIcqxeje/YYDF3068V4z1yynk9q2KRXYdERKQghaWDNJlbdBhn/fmsam6okOtUnsXjE88qGPd+LVHWfK0EfYvYXiAoOaJDK8d4EXnJ4RKJceQPBk8ravuQtKVwnRIr8WtmILHuoztthfwqb2WvZ5iMDt6v7oQZ alfredo@papaya.local", "key_options": null, "keyfile": "/home/adeza/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "adeza", "validate_certs": true} 2019-12-21T15:38:57.442 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\n', u'name': u'dmick'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/onMoOCUhkp+UiJAq+ye4P70sfeV+uMhgWKLfLBh/1SYq1xxb9bSmjBnETO/9KvukRhzlReBQ7y+BKRhm037P7LuTUY4r0Y8JB6BXkjjCY1yXZJC6kvB7xFsvyiDum5rrdeWMQ/Rhh7YJpze5KY1nHLkyU9X7Ewak0kLEyduJEKekbsOom6l0wZ2ZYrBEKFWJXUDk5icUuWWg3mhKaKnLNWVg1lUw8jzYTJYfjR9GUgzZWXtzkAaI3ugpLOlQ+KEls03/WAFcWA9F9ExqzIsJpvEks27LeFBGVE3PjVNKMqqBZQlZtIEz1iguWJaWMi5lDIfXkvVOFabtIpX/BZRf dmick@dmick-MacBookAir", "key_options": null, "keyfile": "/home/dmick/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dmick", "va 2019-12-21T15:38:57.443 INFO:teuthology.task.ansible.out:lidate_certs": true} 2019-12-21T15:38:57.734 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n', u'name': u'sage'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJ4z0gLwjexgyixdTuiwaxpPnq7w4FOKxHsQKKAHTlKRu11ynB+9APbtqaAYss8yDqHaHyewPdE+9JMYhZX8rUgcKv2H7jBCQ/Enfl0UyY643VQnuG8lr2Ipk6ELuL5NuuTb7Npo4zKyq/0649kPD46k/X/qLuKzhN49k50S3XJq5WwHi6CJvR8DzPOkjied0o1BHpfcCCIMorUcyfVxqmeu2m+uSE15ncQsd8BIvH4/YNoncUGYxqVhGB+FNUlWdhV0SPRW7V/gmyvFW5qdYX2QhNZAQHChdEH1E9AI7WiCFkyLRx39qYgXgqj8vOOnlwHgy/oJnpnu/Mlvj1uETX sage@montreal\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCVPfoCpv2/9uWCwEo4clWt3i1O8ZPpB2FcQbBLxY3iLQWZJDe74rNWPBj0cJVCytTHxYV0e5HxYyChOG60ca5odHlPNB+/Z4WLU9LVuHiCYsElivJlwvD+l9kzI+UMGJb3HCHqx2yIRGXpEkSMFR1+w5Oz9sInPD7JJmO8Hb9KkoeqZtkJL0aic/M5bDWBut6SSfYlmh8qnQYbkCDiXg/KtOcy1ZJXQ9ZLjsCbpSzeBNbdUOxEQ1+5vUe00vHpl0vOKtRrzgTne2ct1IxbSjdL16ZgHi5BQAREXc9UUG3z7NPOmnwkuKkkpyLmwcOusot5tYHrxks+5i6EoGEJrvJ3 sage@maetl", "key_options": null, "keyfile": "/home/sage/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sage", "validate_certs": true} 2019-12-21T15:38:58.041 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n', u'name': u'dgalloway'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBAN3MlkYQNsm1uB6uJQ7hWJ1Ro9Gusrhi/b/KshPMFh51VWBQn71LdCCjdrsE6FWs5FYE/hr7MITJe+si7aZCmuf8U9tc+oiYh3spXtRhJVIM80FOVRXUmt2XR8I+oNbpcFDayYu956Volafksk3oYjsO356swj7+Q+AUQVF4bDJRAAAAFQCkB2YZKItSDL7XfAl6ZmJ21vqUMwAAAIEAsZspNIUwssVJGO4r3M3BECWklWLEFviIjGVKdPWulN10eRCczZrH7vMOs8coHKDQqYdS2RejlojT03FjFAqeBiTeAlcmgHoev+vqTg2Yhqx3gROzNc6NmG887owCOU4j7LJQP6SX8I8EpSXCMNOZliv2ZokjX2DuRsuIDxQdc84AAACBAMAp++fcqmGS7x7ic2x4rhyiYmdqlMIl8lrM3GkVR6qGuIVBvOGl38ieJKcTBLeRgyC04uIMfk8P9lLvZw3UF97qH3V0imKfq8aYnBGxCv+IjtdOrR5k4hrUW+P0qipvU+4dgHlpqwTKB4MtD9uDxvhENWvqcCnOS5rI6dddaZT6 dgallowa@dgallowa.csb\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZtt 2019-12-21T15:38:58.043 INFO:teuthology.task.ansible.out:Edz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50", "key_options": null, "keyfile": "/home/dgalloway/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dgalloway", "validate_certs": true} 2019-12-21T15:38:58.327 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045', u'name': u'akraitma'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/akraitma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akraitma", "validate_certs": true} 2019-12-21T15:38:58.620 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e', u'name': u'trociny'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDSc2rWtLtZM0CBWwaxhSfd0T++ZTX9c8kQrYT0rsQlJ6Hxj4c3+ypwh4iivdtFpyz4clxWkb/wimi48PvjNyZdudB9BeQUBeFfLiro3LnFEbKhM1RwRI89s34mRwmC0LLUZwcdX80zc6/fKuPzvBzR1PiY9tWRyIpYErQAHa2jEOE2MpFDHnlbXdfXFh5Tw3OmC1qBlQvgcrivA9lVaVlf+hfwmOQwXFIG3PCJLNlXDshCSUGAfYKBRvQaKS8sOf9rQaflPrwqJ8hDgVVQkpp69yVsGtahbBmjs6QS6C8EkGUFQmvrIcrQ8+5mRlRAmK2DMTP3cAcsYcMOJ+9KwP+H mgolub@zhuzha", "key_options": null, "keyfile": "/home/trociny/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "trociny", "validate_certs": true} 2019-12-21T15:38:58.631 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007', u'name': u'smithfarm', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "skip_reason": "Conditional result was False"} 2019-12-21T15:38:58.910 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n', u'name': u'mbenjamin'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANarZ+C62DPffRIhaCuCtXLChrfqwXgWqGHOLLkmPTtZ3/EE8ZiN+LY1ZcmP939AxXi8Oe+9LepwAdsoQKZctlC7XToyhjIk+rVSPyn3zl3BTC2AxfCKq2vqj65a1sc35Whrw7OaUa2fr5iHSazP6HuG01DhJ9l8HJsLKmiwWChZAAAAFQCNmunc7BNLWrqyrYwINTfscowkbQAAAIEAyYERcunZYJAZ9Eno18kM+ylq0jcQmnPqyIvssWXTttUZnaGQvbVxABeyr6ZKuUyZQfyetofquOGBKPBuzV/7posl5ShDuAcxbS1zhrmk9glTA1yFLvznDdWdLOWG06tjOc78G2AmldKHCdPsmgh2D7WyRr4QY4rcCvwKfZ6+eEYAAACBAJZrOlOlygfPZ4ZB0+EbdYACz1+L/iYNNAOBp51TRutNceLVrNJNOVdmCMLuF0A1go4kPSD/05aJOkOyW86gYLA5DXaW3961+kPInOFAe8OTyQYE7221ThYHRnCfVFjdknuCDomj69wk4SsR+ZPXvSXs1jSJWKhLfUSe9n4bmw7w anet2@secant.private.linuxbox.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5CMht/crEocrHqGjf0sxE9AhNcW3XBxwlniJwaBJLhekLQEQJoxAy2xUUmPYb2/e7rdmTBxzQipUaAGWVk1FHANac4cbgXzNT+9cnF7EIRkPa/x9jaKNiU0ozMhu/GOoeDXMrHIDgVGWPF3Ag/r17ORpYnuFTjsqoCWyh+mqskV+lmAp04rv2tvqouHtEs6gLebKNtB8c6twhrBKF62Ie+rSr3r7hQaqiZitHbg/isbU5RMBl6H48wHAVSK2/uT 2019-12-21T15:38:58.912 INFO:teuthology.task.ansible.out:fMorbnuEYRG8HvlnPRvQHscJYKiZSt/G7XqZ3x/7c4czEF+OpdQbAgg46vh4fNVMxZk9KHwEkOE06ROuwgAejJ mbenjamin@duel", "key_options": null, "keyfile": "/home/mbenjamin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mbenjamin", "validate_certs": true} 2019-12-21T15:38:58.932 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01', u'name': u'aemerson', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt", "name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01"}, "skip_reason": "Conditional result was False"} 2019-12-21T15:38:58.943 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78', u'name': u'amaredia', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "skip_reason": "Conditional result was False"} 2019-12-21T15:38:58.954 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5', u'name': u'tserlin', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "skip_reason": "Conditional result was False"} 2019-12-21T15:38:59.245 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28', u'name': u'dis'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHk8c8wUTTYolOOmMlniNrsjfRyF9pCCnaEQ2DXR73P0hbFMme5Md5cF3PTtPIOHSNkWk/qg7s6gZD8LGyFmloO42RBgLsW1qM8RWTDbPy5EQdrjxL9byVzBBCDoQPIS2M+0XnhYa6CBj38Sw8XPlS41CebAJEOQeyLG2I6CAI0rTij6gsqDfALbIezd/eWvmbksKYM1Lb4Gvj+k/OL/xDZzIZHF/P+mK+trY45gsUHxKZamkeHyAT4f3kiNJA+UwzTWGhafy6UE7yGqhZjt9xOZSQGuwroL7EtAzntxayPRNxQPfGDV+fYYf0QBbTtHyJeikPUuw7aR1Z3KwuSmWD dis@zambezi", "key_options": null, "keyfile": "/home/dis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dis", "validate_certs": true} 2019-12-21T15:38:59.521 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n', u'name': u'gregf'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANaS5gZUnIp5UL/PM3xOrQiNVxfona7sD1XEXGYvNUyoojPWWAvdQRwdA/Ay8JsAjKeWUt0IwfWaPfbEoXU1TKAI+yyv5WzQUZAKmBFuYM0CUQnKrk72Dp6n8P2tnYen0ect22xEiW2+Y0JZ3wfV3IB3BQPoVCC9FQ+P3LHhYii/AAAAFQCgSceiZFeNItwJ2rJeQgVVTQKljwAAAIEA0Gd2VNCbbQ48vAF+m/i/CeQHJ9Dom9IOBN3v8XnSlOhtNzY1pJ0NRzrFQURCCeVQJaywr0et5FCXIDVqTuR4qfCu5XXU1KHpn/AvHstOd93Y0XYMwSXfNszOtpH7w9SEm7uBfKnLQNF843/MHZL4ZIdfxk73YjfMjrYFcvJwvM0AAACBAI9Wi2wG9Rrdbrgk8tqbnKMxxJCYt/Vakn5mQSc/WHqMUn14+Kb7oBtfTsfPFNK952FzU09ptNQvNWTPNMniI8l8lQi2B+Z28XUlFLL76uxJXWko99xFpf/d1kFBL0M7GRGmvvrMfeTCI+WIqh2jBVqbl4o8kjNx4p0VWyoiWeFJ gregf@pudgy\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAwSe7szyqMTqKK3D/JsXIa1PtPf1p8Njk0sMDSsJPA0gBGysmSgstke9fglOgeNIbNOBNDOY3vltuNCgUiQgVWOYhUaVpj+ 2019-12-21T15:38:59.523 INFO:teuthology.task.ansible.out:2vtSoAWpXMQbNaWPGKhbxvJWMCLVckLENyoUZUBKLmTvRyKDjjq8CJ1TnfA2Xfo3a796EIOtD6essU3lAVIfIUILGD+HqptiHSB0spUJUVRxxlO2xw9rLYij1eUPRHrr9xWvDamr5JtKVtCLOL3M9QlltHieKlbJSCZq/EVxW9NAGA6IiqOhw2E1k59oGZUq4Qg+YLgYkB80Ms7nb1xpjQ8txvY+73xL/8i7Nqmr7ekKJfyTpRe03nCQ== gfarnum@mozart.thompsonian.net\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzqJQfdAKaggPDgSpk8P6QjLeyQYOz183fVSKnqAa5EeaCJozS1dn8z+iphswT3YL6wqIMqVHMfkSUcEeTBwZmvydaghOMJEIQSP1PCzH0V2XprELrEcBC3QTvaElPjjEYgBVUvEw8sL/RIQ7mgt9vxD0NcCDspsz35fmU222KzFxD41PZ5XzHQ3Lumwdax7mcpqrw8Aa5/tgcEqAIu54HYRf1Qmk6+ueMAvpwkX3eiSEtkVvuV1Kxpc3/0mOfJ7cn+eyPX3qUbo4oY76hIYseoUIwEaVE/pNW+pjwDSy4gSOcuxYDEGKzCwZPZeQtRQWzRt3YNafxi5xsByobGLlZw== gfarnum@GF-Macbook.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1ik3LbyKsPiGwdh+vGrKSYLvhMg7OkqMwDNYU5LqWBSotogxyGQJUuD80ZavxmmdXC4qScddHtWOJjtyX1CTSDUe7INpSgGPGnmSExVFpcYRnmxHAB+9M/c9VorUkQ8IOKLmRO/BdYma3Rl9NzhHI73yd4zJlx1K4NuQnhnj0Z4yXO+k44RkBWDdMicBB5lBP0hETaH2Iq1u5Dai3u6N5N9bSPXv2PGKvvU395K0Wts2CxMhOBc7LAixysNzNWLrnG1ghS4jPVG2TIHOElP5lIXUv1Wlkool5rTUWgnVY/7loWkDus2JryDM7hLGFXqgT7HHcpI4gSITqRhWxzsUF gregf@fedoragreg.localdomain", "key_options": null, "keyfile": "/home/gregf/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gregf", "validate_certs": true} 2019-12-21T15:38:59.813 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n', u'name': u'joshd'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDgS5EfJyOxBkl/TtJikfb38rm6muQo7MwYufkLSBw9W2jFzt/ZMbVXyVsdZM4I6hWfZm1/q60UcER4yoA1xwYGOTmbFXrLUqqKxKTnCdyejuQ3miB7aiuisWO4vjb2+ppBwL9Mx775DPeQJehFyFK0qgh33tZO/9LWWYy7NOZdA9L2ei1PgMWa0VBeijcDR5GQSXzh0Yi+h/Ezu8jevu+NkW2IUJF6zPGjd9M9lAy7KG2dQjqqCX4XWmUS+MHJS6f9bsC5i39GWy29FvX/qFEArOMzuYRIFWE8iza95KSBxptb+Hw8PFBjQ1kTh6nC5ZV2 2019-12-21T15:38:59.814 INFO:teuthology.task.ansible.out:X3ETcoj8fToTcgW3dTRF jdurgin@new-angeles", "key_options": null, "keyfile": "/home/joshd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joshd", "validate_certs": true} 2019-12-21T15:39:00.108 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n', u'name': u'davidz'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCyOWnAdvzh5zR2ts9e8g0wG4wZ0uNr0YJUCncAOTJB0L8xkQU76jWzX3BRNU5Wk8VtLvpJAQ61P7UXyzgGWSO9FHZjD3RD 2019-12-21T15:39:00.111 INFO:teuthology.task.ansible.out:rc2V9uh/lqRKmx1COtDvcart/JwzwHlsWdrYdcXvEaAiFrZJ6x1jJAjIL6tvMO7bqBz4YNruKpGWsQ6p5P/M5iIEPMj1oKanLoULXqReC6eYFTw3LNAXjberPY5GxYwaKS0HuzySRvSblShyA85jawTSlEbsOSnMqqGu5vGJiC415UFQQ5qKBtzrgAjlUgMbaqGTnTmBYn7L751gIWRetgWqwuHgfIe2ZGM3mmh0DX8E03DUbEt4UxyR dzafman@TrustyTahr\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDDtSwGzH7V5jzppggL0+nYlBSM1HqGjkInwxutra5iEyFoBuV73MNSBk3HrvIVGN13+PCdHf4Oy8ywbhC/4yKbxS1NG76d/GU8ijZFuBlEwxwHZpcThpSTuYrNx7FJVp8Yjl45DtD5s8FMI/0YZZhLlIqT6QCTxlTIPXDb6zefEWvAVziE0bmhRpe4B6AOLOGTB9fivQr5YzxhfaSS5Vu4U1x5gmosouSJ+aMFrQWLqa/eoCy8t9PnxdMjlHjHCZ7Qv1JfpaAPw3qOK30GrVeXR/hPN3aFodTuIJpzdynjk5HO6/VKW5+mWBO7pmQqn9LNQEjADXbLh7PG//nOONbH dzafman@ubuntu16", "key_options": null, "keyfile": "/home/davidz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "davidz", "validate_certs": true} 2019-12-21T15:39:00.405 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'gmeno@gmeno-virtual-machine FKFu8B2pMqotpmEVAO1few 8229574e499eaf767a408909f5afdf2e2a0bb8f3e61b18d63a651f7102c68dbc', u'name': u'gmeno'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gmeno", "ovpn": "gmeno@gmeno-virtual-machine FKFu8B2pMqotpmEVAO1few 8229574e499eaf767a408909f5afdf2e2a0bb8f3e61b18d63a651f7102c68dbc"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvWnKfzTd574C4GXg1mr4pbJOInKv4qMRf0DwksTzqtXStlhT+PaO2C8p48U2Xjz5hmPK9J+iHA2xKwI6lKI194wUo3tW283euRyTCN7LlbV6tgD9k78hDaeA+53MHYmwbxjHEqAadqrLU+68LfmL7gaaUSToP/Ap+hdcDC7H9lulMcxkUeahEiDfG2s1mD8IUM6XVuqZM9WEARKpF1oSEBKhMMob9YJZPpJWipuAGkPvDA0yHK1lB58Yr2x0yaezVjQc7V2dwaY1gXR6AscU5WzlwWNNwXHqPdyyWRaGx/9Fk00yfzX4V7ZqTvKdMLKwB7EnamduTMPsaoKBUBJCh gmeno@gmeno-virtual-machine", "key_options": null, "keyfile": "/home/gmeno/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gmeno", "validate_certs": true} 2019-12-21T15:39:00.683 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1', u'name': u'ivancich'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDkIZyWnJXYv/z3EVeKHatjFUAdaomIMqzvFLrA9VskeM0Ctv8a83Wk3BbYvTFQWALjZjS2HVYzi5p6kTQ4qxXlcO7HxRw0j4otNauilvgIc4Lu5N4pUtYf0piEo+nrXz+WhELfmJ8t1Yx00uGEl9pOF04SOuXtpwwrCFAO6WMCcjDuYhoI2VBKO3W/OWcIuQ8ZY3j4iDdofFOUGyyoYVyOi/HILKjluYB+ues5+wBEkDuoi3MafCnIj8M0O5fU8/JoOTksOu779UN6blPiNdEgnwD00jA+RWYY6b12ZVkiF4rUVGCiiKKUwOOWkRhkA5OEcBgZ8cGWFgGfZM1GJA// ivancich@sleepy", "key_options": null, "keyfile": "/home/ivancich/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ivancich", "validate_certs": true} 2019-12-21T15:39:00.974 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n', u'name': u'wusui'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsTZI+b02boZv8jVLHQoFlloD2WF/3xT+ZfZxzwddL7Q6eqo7j8i21LAbLAMWXYMyhiClrloDeHfCZ42gG5bqSThF8LoCw+GspF6SahTI0lmptGu69/OeHqqCpW1DnsWB3wWqfi90XbQJrpg3cdRaqOCmkGVJrqfFWzff8Zg9IuojIH/aO9drCTRGa1pHHCyEdW9ZNeYFwrMjfP+wJi/mC/ABAoXyDy0NWT0uIA+36abNg2/ptok90rdan0r72hjFEz/1hpupKWri+VXAepiyDMuSgJm9uJCXDFaNTn3jocdX+sbVtltThQ2OBxe1T60vRLbXx3lvhxxZ1fUkcP/FF wusui@ubuntu\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD2UzyfBk1nZ8SLCtW9UjorMeOo4u9AM/YL8ZI74GdprR+odERZR17/QfwT5vmLVNa51+CQt1HcGodlQYfR/8n/OUl/gJyVagBuU2zgmLPO3tOor5RBfms5MZ2VZ9Vm3+WrCO/vrAq/5HVyGbQsKiQHXkIRSL2NIgfKRTOVTNmuTSXtZNDgdWGIBmcOFQv+Im9CRopoI8ZtJLQj7xfxtGtUas7FbWTcf0o92+xvWuGWq5WQgmZ8MUIfvBLYLmicuVfsWjZ1YsC1PNhuhkwrW7MhaD71Ej3ll243tG1QD7eAmo+psIp8phlU5s/5FeA5y5OzDZ2T963oHGhq018TYO0b wusui@teuthology", "key_options": null, "keyfile": "/home/wusui/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "wusui", "validate_certs": true} 2019-12-21T15:39:01.263 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304', u'name': u'zyan'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zyan", "ovpn": "zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCtmxLK2rSomB14d9/fkAshmsXQfd6cJ+vEb0MZDaUT+OLmr1vAq0+H5g9w3hApJ2UEOFSaiz6ywgBAFeBeu/KmUALOhJIKKk0imoTunYclqxVPBK6tG/2b+ME8VE1afoEpMem3Bzfkrq0pDt6DLFBDlt9rRIqtO/ZlqEayWkEwTXJ7pwj+oJS03Z3Y7q0Iqw8PToKqJWwy2OvW1I9Pi0Q8s4YV9I34rSO7Wv4w43AXpFHRlvJWIcI5g5WFI5tnjel+kUAWMZft9ynFoAP5iXdByq897r4jAF74hapaxLMsHBz4MyYmPZqS76Cx2e3kGqXeetPinfcZxC3ViboR7o/1 root@zhyan-desk1", "key_options": null, "keyfile": "/home/zyan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zyan", "validate_certs": true} 2019-12-21T15:39:01.543 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH 5ivdxgFO4eIkbXVhl8xkvw 59212d29b8b42d9fe457c1b2c43d774e1d25807be10dcc1252d4aec63b97a467\n', u'name': u'yuriw'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH 5ivdxgFO4eIkbXVhl8xkvw 59212d29b8b42d9fe457c1b2c43d774e1d25807be10dcc1252d4aec63b97a467\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCw3MhMNDYjkiQADoC1mg9ga0Yj174FHIFt4RQIC8qK+6cB0N3CM5R4JwP8pR4tYkVFWmnYRScYK5VVdC873Ac2BzYxmse97zLJTbUnYWTdY+OFi1FUozsy3PwtiNHSd48Z/PBvkinQIey2QlMKaW3/yz9Of/Yfm4prxg9lBsd6/tOOFj3AsqIQ1OVhMfgeVEkdCL7AAbEYyMxePZ3UFrDwj05HDLgmFzx1xsNJFOuBwLfU/QNHoXJQrcVy0rZq9fB5rg74iz1JZs0uDWocQQK4Nd3Rv+wRGBuew2fmkrS2bQWXzMoSL+/i7pbf74LKcbWAE0wXEImqQeREHxq4ZZ5J yuriweinstein@Yuris-MacBook-Pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJUmq7btpBMjjODHXp1888xoBuX+rhiZg3ufuo69RgDxUMymJn3CPJ/5NGXoZ3SGAWKk0w0Dx6K/VO4y92mAV+cO81y8wZR4J3fPV3IbWRjN0X3IFoAvJi5n9v5hpG7z3RZCGmQnOZ99m1i9dge/3S8NpuPLIvYkSo9z00d+mNP6D8rGnqOmEBfEhUQXLeH7rbPwaMWPgB/hrpiPCyLR/+sGC5bq9PAsKnocqZ12cnvCtYxpnA7et6nL15ftEpUrKYAmRDxotuoUcsllP66GSszjbrgN6OBNDSqpGt/+RSJJwzZEGQ073gJuMveKy9tvD/PyEttjvVKL2KsBuXGSuL ubuntu@yw\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmXWgRRes8bzekTH2fTcxdAAquFn1SjPhws1 2019-12-21T15:39:01.544 INFO:teuthology.task.ansible.out:Z9CkxNaqKe25XnzPehqrhg9etouUwvORQHdDJdTlmhTSYtn588Omq+1bvniab0KzAYgGZsWdPma/3E9qF7ChXIMik9Umip0drzYTaexCuC/Nu46brTxlnL0vaxnnVjDwmHC0CEJs4qN4e2wSlsA0uBDRWjxWFMdTUgSzII5xasHypUYZ0y3MNdbfY+SVF69DEDH8EOQMGYm8JQXNJFloQvmvLUFqT/63FsdpzRVTph4A8hvI1f8i6ULb7fPQN5ZS62wdXd/KvrBXrCvhOi7R63RM60mkMRXm1XajTGmCKiikfiqgTr yuriw@Yuris-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC41kRr6B/RC32DrQKRByalH832H6BpCQF9XdTvgYi4+LJt+kAuIGLDF4xm4uApTmWRrLspWm0Kksc0ger03uNx5qJqMEUnJvjxJnSDC0/XyxWapVxIviFkvv0BjbhVNOOaMU55ywrZ9hHgMdaFwuMuCOaB7f9Tm6IUXm2k82OXueeIG23Kziq+IQBysWOjvNUy/LLpl7l4+xOkhNUupUiQCqHvFH98VUdYWkHeyamafGVZmYbFLrOXgmviKo0u8LPxcT7psKLk0yFG7fG0DSak6DDdKJqb+G/NqMSPxRSAI4FbXQ6UOZb32Ozr1Sql9yfOXEYWZuatn7W5F8DjSv3p yuriw@yuriw-RH", "key_options": null, "keyfile": "/home/yuriw/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuriw", "validate_certs": true} 2019-12-21T15:39:01.830 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n', u'name': u'tamil'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxrS30JmTZbnBd73s9PGZYxCkWH3u4En9Q0bYJan45W+j+y5vMzaNUVsp8We1tphqPQ9B0ygi80OxbHWu5gyv4g+M/JFtrnxv3QgBCYWeBWfh652Vbdlk1thl6btFnHUhVMSn5tkdnFxyVDJHIROoVXkHRaCNAUUy7tB60IhPAMomNcCGu9R8+opONFLXKTIfkYQ1cGbysr90DMs8ASEkvxdv40hIWhK4ufvL7CFFtgXuoC0c1WedBFOkr5LDegzL13xN97K9o5J8CAU0MUKk6H8KNKbQuNq1zANwymMFY599GIlxwai8iluJgfOObDRdCDmKM2f9RAEUXQM7oYoH7 tamil@tamil-VirtualBox\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvwiWVhNLeOLg72cpVIwbUYnmFh+Cqf/mE7Rc/Xci91on3GPrq++dC8t2GhTB5Zyq46EE27SmUkiZpHRnz7xwiJKM/tTgNEkffXDK3j0pA2E22iqV9FpfSA9Zdi1Puq7CPT7/rix4rF3SPvMtg/R1fkyornLFdzhJiMExCKJXGwf+QsKk4NVnr+nO3EEkKOoP1wio1bceZ53rCt9Tyu8NawAB2nL2fvtBPQDAriQ8bLX69XLbOfhFI6GJo815/j6CQxNmvA0l5ABJc8+A71jZmndzumFRgqBuhiCM5ohsGR0JGahUQjFEA4quNTx8EWTGRsqhAYszzpqJzuN+Xb6tJ tamil@macbookair\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEPzEyDGI7poyXVPG70vgQpvlU/lTt6Vvv5CoHWJoSrHklDdJEqekhCB87BX4DTrqbb5xoM7yfxle/z9UzgVwSUWqsPdZHA4EbrNFRKFxi/gc2YrygVSjumTyzf9WJJ5vCZvTQexOj7rjWVnJkjXHKMXZXIAQN/bJf6J21t+4A3XLrl30/4XXyq664NONUNjBHxHtlbS2LM6Th/hK4B23oj2BugX2j/50vWgd9H36bPr8+JGskOi6MH2ndZv5pYtLznFzQmsbixjaN8kCuyQdlRCOndPEFoHFhDla2MeB+h3XQLDXW7J1TR2URFSfcNq5ndDS4i0b+t75Fjhb7AtLT tamil@ubuntu-delldesktop\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6tJ9y3solDCzFdXcczO8GJEwMqMYXWU5kuMz0rr/US6F72j0spSNmwGemMOZE9s7KtHgior/Ufw8/vMRkR 2019-12-21T15:39:01.832 INFO:teuthology.task.ansible.out:hoBYB0C5KT9utGNqrtDdvH98h43XnKZ3K/yaHv/T+u2EYWNgHtOXMtgIVFnGwSD7RzF6TbwSfurprTV6ZUyyOxRcmHnkmwpbNwiwIoXB03k2PITdYCC3lEBEY0OC+vAW/E3+bFbV3RSEA0Cm31jmMyZyPeYiXiEnPXUIClsoDj34NwJTIgctwIsqM4eYUBg3pHliWYKEWDs55HZYoj3W/6Jw7MASzyAYFSmP8rQu8iS7tO1iEBRGOXFNxHirGQ1WqUn tmuthamizhan@MacBook-Air.local", "key_options": null, "keyfile": "/home/tamil/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tamil", "validate_certs": true} 2019-12-21T15:39:02.112 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n', u'name': u'jowilkin'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDYGVqKZgLx2LJ49x0kGo/Z5r8MVefSUCYhhrK/DK8ht+YK1AqLXErjAedgpVz0WuwirRjh5YkiXGMYiOdkMUYWGk5nwCHEKnnCpne7Xh1OOqge/QGHlUigWVH8VIHBsZ+wTW25mqOrgDSGB8+Qk7Dr2o1sLGUHcx7vITDWOrer4KjF2+H5TQH/7irUVORu6fzeGKtGyOrTwQ+ZT1+fuhfznDULFGyvKGdcXxxNOg+21Fssk2nCNTlgRCGjHSrfelfuMT7QEt+9puWd7MWXYQg7xx8jkJGMpEQRFxJTxnfUlUXf3x2lc9Q/SvKfpVMtc0N+bk2F9+euyvRvjtn4LJNx jowilkin@jowilkin.wilkins", "key_options": null, "keyfile": "/home/jowilkin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jowilkin", "validate_certs": true} 2019-12-21T15:39:02.401 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4', u'name': u'bhubbard'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDP/yVttBEs7mpVfJL+reXb/BfB7DsLrA5ArVqEJjySi3NOz3e/wE8KRWbHD1UdraAU3P8b96GdAQxGOs4n0muOMejmaXWZCP3HkRL+rcd+03ZI1lb6ga9XjBARZfb2h+kg5jTJp/Ll6mwAYbSXvQ4MuDQzwGRDX5Q5Xme7OZ4Wb+vBmmkihHydySyP9YXbWWsBw4jvZCFSKMGsUVy/HIt1coZx/PyCG/K2Kl0nrE5ux2QqfdTIpzYo4c3y6uMIHDvt29lRmk3Gnh2//U5eN8mmRRKJlkvE8DKy3VYsl0JsP9pTWUZEYgV0EX251WTO0akQ2bKuszqd9Ai04RWnXUu5 brad@dhcp-1-165.bne.redhat.com", "key_options": null, "keyfile": "/home/bhubbard/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bhubbard", "validate_certs": true} 2019-12-21T15:39:02.681 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n', u'name': u'yehudasa'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAyRDKb+InDsnW4E/S12Pr6rK3wDfUG3j+iw/NsCWwN5xpJEA1/NhVlRzY8/jejc6ddmnNKWbm/UcYVrbo7yClRTGu45g0Qm9lmfvgDgi1k8mwUCy+edbQ/R8FB9cIHHCnUuWzRmE2XGaeDiSVQgiYqq3mOPY/hwe3G7jwFalYPsbmY1jNqpzODmKFMRd1bOwdHEOMTfh0owHWqhhXakU8em30lYraQVYH24FplH2NUBArxmTuctyfweHxx7yH3ZFcXFUpUMSLuVDWb1A7fMiL6eW6wiwh+dD+q5iFNvuioZITP2EjIsb9nS+eM9bOJZuZ+5pln2EsHee27LjunbwgWw== yehudasa@fatty\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDF3PxJYgI3tS7SG+RnUvxMxr0/XwBAio0oiP4AC+o8rUYYyPd/RU27TR1kphbuJhQWVtaZ6kFSBQeNQUxv6rb8hmMzvlP1FDEoGkjaOHOuYEc2DFkkEAfaR2AlNKn3pUQgLE00TLHctCgd57Cn2Ppt8xEi/vTuo0gX3xjWOL4N9QDElCXKNdxdyyJ/RNb/Mid+YxgKKjf8ppmhYsQ+Qqu4YwPzLkfK1Qb3YtR6Hbjb4jxC7C1Z13h0vUgHe7qSmUCYNo9eC+lEgNmmnLb2sXJaUUJM8MEex6v+4muWM9hV0cq/Wplrx8fhMWMa978sxH8yVkE3Svqc/K4ZWN59l3xT yehudasa@yehudasa-XPS-L401X\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDvplQk8xkSXF03uSJh3H4zyZHRMjvgUJBgGN27Kb7gAlIQv306wPKIprOwp+jDhEcPnEWBMRJMUJuBi0vEiqmDHt8mg8ZYcs+Gi7MbUgBokILdJuSG89KAgq7gBCuIhYbSg35pfvXvIZswiXvySqMrJNB495GYj/FrcWEBfQMuedMSkIfNzqPV9yR5YRh8fnSs3fjCzV96vzu7O4AzgwICbM6XmqVLXPax/32Hq0dz7sAawlQFCJYPHzVzjwm 2019-12-21T15:39:02.682 INFO:teuthology.task.ansible.out:ih46TRJnXrxewsRhk9GwKfI4SWaFIRdOdadGFSLgdZ+2iGE7hTZ3sRgkLOj+4zKa+xTbtjjXfw5up7J3hAmrK15Tz yehuda@yehuda-940X3G\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfNxvAw6cb33jg36vw8kJu48/BttmFJ4BFeanYqwRgtWZrKpAAGHEgCiNeyIttMrI3t7Spcdyp6dy6n0P3WUFVgHWEYtGqXwXqDC9UXISkUJVVDZoEynpGC92RghhzH53+7jX1Ytpr30r/xb1ktryTPCoEJRqyrg2j1b167i6PV5tuJCA+77gW5LWBmMXJCprjnfnJ92Gp8ZewBB6gL1x+3RFkMxGsibL0Qw1CTKfyAHNYc/jos6YiL0b42Ha+NmKwvzZ1GgIleRMleisSOyKPUlPdR7UnmtEXa9bl3Gc/hc7Zt+v4iUypr2WL0X4Yl7rJNH/AAd/Z6jPkAX9nHYzh yehudasa@yehudasa-desktop", "key_options": null, "keyfile": "/home/yehudasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yehudasa", "validate_certs": true} 2019-12-21T15:39:02.981 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4', u'name': u'dang'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDx37xbP0jYqopygG66LbSMRPoqpGUPzQZ3G8eqJCk2I dang@revan.aa.redhat.com-2015-07-17", "key_options": null, "keyfile": "/home/dang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dang", "validate_certs": true} 2019-12-21T15:39:03.263 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb', u'name': u'branto'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2YyDoeKdaCuJif79AxJAorqpUj33WHbHu/eIWpHju+iIF3f94K0qyEXgs4AwZin9pzfyd8N1aKNT8ZFPss6L0Xk68aO/6tdInq+1lTMMUC5v/LBqsIV2btPTFHrQWA0R10v05aZFvFaSqhHNsZaj98vQV7P+jiSs3yPWqCIWUTQ1VXnCeQnB4R5tob0gpMuonhkKq7vrJPhf6ge9Av+yjL7+8oP5LJqFJ5npDxkwZ0u3L0QQ9ftAT/e7Y7ESpwpgaI3Trl0EwAXFmiLjXEUA+FBbE7ko9kLqXqRjc1C0BdRG9sO0QKBGDBEx1W3w8rCPKgaDlVyqsBT1yvIjqqXDOQ== borix@dhcp-31-190.brq.redhat.com", "key_options": null, "keyfile": "/home/branto/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "branto", "validate_certs": true} 2019-12-21T15:39:03.546 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6', u'name': u'xiaoxichen'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDO00mus2XPX/EfUZ/YsKI84TnWj5TbZP7Fs01EkoOTPgNoYysVdf0PwhAooz1qPrv+HUsUMpQT6uNYfCHyqx9rScR+XUlyQuFe7Sjp/dn3y+b1PKa6G4HX7f1eejR3ZaLZXA4R/8oucAXqQS5pDr3Y7prwkrp9cyk8uawBXoDcAlA1SMIPZPrtO1+iZw20NDhixsG7NE5jMFfw3q8xTckZEqBbteybci7X1v4lEYXCVlbSbUhyZ2Af8HEj09JBDOL2bI3yLvj9a5Oj/HtKubyApzZpA7rgeoii2UZQGuSRI1VJktp9YC0zGk5S843xYW92utbALKyQ6DdB2F5gc0DX xiaoxchen@LM-SHC-16501223", "key_options": null, "keyfile": "/home/xiaoxichen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiaoxichen", "validate_certs": true} 2019-12-21T15:39:03.828 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e', u'name': u'ffilz'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCz7e2G3Ziaus2yMviKh9s65e+yaqy1+ezs5IC6US90/8WumNAWqC4Yr3ckvCc120r3VwpJWcvaAWCF+4IZeO43lSNLadbCnJONed5YklcUuIJvjC1eDbZTMLJ7h0ijgNXgAFmFq0AV2NHuE2L2VKXk2hsIqDVGjZLUpi7W+ic07xnScEoXXDSk5mhJ6bKmFsUYCC91g2roYs9Ui5fGd+ey28SFOADI7osDLpLkiTF2OzGel93o5qobNZwKR0AxGsBIta17mvf3I3307bGnvugDXpNB0FaS1QZhIzMO4G4LZL9zahCvv4wm5GmuO4F0+Ac7dwFu3KZKtHQe3g6WHuAv ffilz@redhat.com", "key_options": null, "keyfile": "/home/ffilz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ffilz", "validate_certs": true} 2019-12-21T15:39:04.101 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n', u'name': u'joao'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpGZK2cWUtE85Bt7fl+kSeiC0lM/EN+H2X3ePGq3LiD4g5R5n6nJsLaJaIZXdJ96G0JSnj07G8Xu1CkVbGDHuVg2O9tnxLhz6tzDwFYNFrQIpALlQwOq8OiwGDDyFkMXQ3JqCCWvyD0HLysOJQIBIlb/o92ZCSAGOyNX1JjrhXF+mlBk1AUW11UeZOeGZQ34MKHapH7T/DuexCz5gfZI/jY9Rg3hr4jb1zGttvAF2eeLJCjqD/Vi4mowYB/Kn/PJhNixeA2sli2i8r6yO/bFmA/PWnfSl+EqD4mUJWb5ATeJ3hyW8qVLOogY6XVu4Sd1VgL7l1RYzXuxJV3Rb1Yw2T joao@sepia\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDV0jbKpbmXW1bhRIOBIdVV1L73qjls7s5pxiW2EMPvZLYzxLBE92e3Vf8bSMWlm1Lyb7a+u5BR85+2c+h2F59sgU1Saek5ebPpFU5qBn8AzYH18iMVSPciByOzwS0NXwWnksckWxUkQgP9bZP/1ORQ7jWZEOHZ5mEQ0oMkP0bfG1cpFrNTxOhDmwbCiQ8dp6KeZaMg4hErALr5Eh/EESxKIkuwQzk9bDbtjPRDu0F2dE0NCjs0k0BXmDA80GksxuOacRxE95K6/er11qst/VqA7o6BfhxJMTXWMdj8NhzIUNjZ8LqLTG6P/XGR24sA2d2WBhlEKRMZNqeEjzmi1REr joao@tagide", "key_options": null, "keyfile": "/home/joao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joao", "validate_certs": true} 2019-12-21T15:39:04.391 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n', u'name': u'nhm'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHhYb/mGgHbEntAN2iPFTV1k8MYDs+jtsJDPE9MWtUZXksdeG8cEfXDBGhiBsprjELsW4+czJBqqESugFMevEysTVKIo5lc95bf1nApxfiRyMnAxb5wQ2OeLY62x26TWJhPu6HMajV+gb8CKjxDobf1Ix4ELbYlL77SOH1VVDtVZ6HiwtYngXAPMONCdI99cuwJlBpv5RjDoLwljOoxEXIh+1Fsvth+iymp0rIPuJx91DzYYgWZFfCLPHRgDy0H7YRdQWUzhrxTyzuT27vsNh5L8HgdowzOu9f2HwGzlTlxli94IJLMBY2359xsqjSswZzAv+BiPDAZMUwJpYvRvSt nhm@clusterfaq.org\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1JAKymiRDyVar5P75cMycxz9u8VsLuNtf/0t1na4ofa4YmanKswPQQz1Am82pvyPqXlBbdIGVneyMrkEoFAk4nTtfmCl7WdtaxnH87Dr9BOLEp+lIPHy7jpS/dcVmKRPB1qCYZ5GNAZ9g+DrXdaeOJdUOpIK49BP1IK8Uw5eBWAx/ae6/12yz+d/6e9kBvO7OUamMSne6Ghe2pRoI8jFOdWMbQy3ofb5wSHZGPNDn0YTGbNWHlfEAcrFmGQSTwohca4SqCJ1NgKnYu4qPVmb3Gbpm+MpUPs8Gy++rQ8wVZ5xWZW/yReIbJCqqVY4vrQzVywQXRucIrB1VyvShuNW3 nhm@metropolis\nssh-dss AAAAB3NzaC1kc3MAAACBAM7BfQAinQmLlkTgt7IIg3nQyyaJRiWA4agfu/HncrcAmepGqPCwy451g1eLMD3AC4QptHnV80Y47YQTyX1VQixIQ6M9zHG6Bo7IXzyqFHFpqnDZN25DaMM4na5bf1CwC9RtR7VWubojatPylOpflkChUBuAGWnzwkz2cxl4Dc47AAAAFQD35l/Lg4eTqZ83eTgJ5lihzHS/awAAAIB/ZdMSE6M0zEvdNPsWY28bvQ0QXN7w1ncVXGachA1JOEFGYvsbHeoywLpSYoiPLV57/wZT2dUzG70u80IQGildZ1WgQNn7C9r2Z3ZUgKmRGyC+PpYEa6nv+U4CHdMViBRXxNrP0bj2 2019-12-21T15:39:04.393 INFO:teuthology.task.ansible.out:CfQvUxUxrHWtOu/da2W23cpSs6Wz8gtz2QAAAIEAvf4vx/32Sy+kbvqOCYz0wu0ItlECN0qOk3gAMvphYMI2sW2ez2+Tsua4eoPRXT10zFxiRTkHQd5FSIvNWXV2iqed6rpwTu5K8vJz2duMIyI+xWLHo4XudtZdNxggkf+RW9JvPOMLMp4sCzBXxpreHBsb1CynndKeYVrgMLybZ28= mark@Marks-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfDOM2v4h6Dk4pMWPQxxHI5rhFtLjkme/4hAp6ki/kJ1XOSzBQdQmD2UPDKvfO9lKO7v12G2Wt+re+0giAcSkfej4+xqV53u7d2pTbMI5U3zohZt8AgESx22DAxj1wBZYuyGO2TRI7DRQMxC47E8EJgcmZBBnMKk1Rz8k71pTmg+CI2vjWx/2x2bup9Mqe4uMA8ubjg2qRlWEqPKIKcHZyrAPdIlpy2FSZ9SPa0mJpdl6aYcFNgonD8BO6A0u2RPO90KmEWrRWu69vRvbbTDiqE0CkJJlAlu3jeFVD8US9vymBHSI0BDFUbukuyG1tAujytqGBRLzxQDU9JOYMP45v nhm@mocha", "key_options": null, "keyfile": "/home/nhm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nhm", "validate_certs": true} 2019-12-21T15:39:04.692 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313', u'name': u'nwatkins'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaetIvR7YgpCq142fBQtOYsJrOMW+Wxu1beNXPJ8qzT5MxD05sdKntVE0DPveUydaeI8/vHeNjXDKN8mt1eO7Lf/rDKHOW/Y/36bUfQAODru9sdNCEJfSPgq1y6+X1ISljvLBXOOl+nQXlx3n5OjThNQbmdthgCT2Fk4hEoK7gn4+ol6foTkTdUXIBam8LuPU6f1KIG6ae1oAVFWLeavAbvmxUTmVl04+hKJWSPnvZxWupaPuiHvsImaQ7GwBjKhzL1wSMly0+pa2zbVYZKkLBA7Fx6fzP7jcEieOOPM1YU0jR5JF+WVzeRFJ1V9n75yOw6S4ef/X/zwWx42cdfx3N nwatkins@martini\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDRivdSX6cbPrdMrjz1oYgn0hb8VQo+QU7wD+AcCQi/CH06ny+2QjSr7bfdpgQ81Wgwqt9+e3KcLD5qdt3tneC8sgSAR5QUQrWKhPejrqdV+3990k8k0KoKIOZ9MGFc1FBv5IpuT/2d535iHJb0ZEg8mFKqKDw71sBEoQT4WElatf5DeVC507hP7GnrhZgtTPcrVddOLkmmqTQtLE1zvo9Y/eel8O2JMImMd6mdVBIcFQ1alU1wV4jLdMpagoPelvVTVeNkGRnOgvvQac5RXtGSMP1R5PoGz+1FgRhDCtdTXRfOdtrlJPtUjYiGPpIoKIUymlvQGjUrrK3WiQIIkIfDtFFu48EN4drjgQpCEQ2D0NMKEmk74mjx3CJDOF1Nj/2Fa1kjECiz4EM7sR8dDG5cW5Y8XsDoomgugM+rsOH9kWFFClHUJmdlNGWLhJnu9UQHLYZjQw3zT0F+FXUsUxKJUVTUdxQx5HBO9QFsSe13NPDnX64+SQ2VVStt8oQAvFO+6260zbux0RVaJM5dTNwnIHD/uI3PeR1z2b+iKhyj/9VP2uBRpGydLw/etQu/lS2K30S+kZIDAp0sqwXH89bdq3AJBipcryrnW1Ov9K5Mfl1UlV95/MivrnOFISWN1F9XR1kuIIYdCkI7hq/FHZ9gd84AjTiERfxRv8VCIHBVEQ== nwatkins@redhat.com", "key_options": null, "keyfile": "/home/nwatkins/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nwatkins", "validate_certs": true} 2019-12-21T15:39:04.973 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22', u'name': u'mkidd'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJD+2BHStkaMJ8RStIIz+yDJCGNT1wbwUnmuO71vNlFxzIyHGvCq3ktx2xTwZz/kj6ZayXYWIRH//BhK2/iK/v+u2WQMBURfMHbf0lDMmNfbHCDBjYvEYgFo4Jwvzq4WHus0AHh9TdVt+TpcxyGPhVbUJh8NZut+w4cRIKu2xGoZZGDeLgv9ijXvDPrCdBxtpvQnF99C0bCTdPM7MK1FSCQEaRmNUgzEj0H7R+gmWi0T4HUD8ZY7y0vWbYPqT3Z07My1u2BfRBYpXnv1655/b777TkSlzSasORhSTT0AkW25u90Rsj2JWr39+49/Ia9uOyFtZIirugEv2v7v/l0grN linuxkidd@zenbook", "key_options": null, "keyfile": "/home/mkidd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkidd", "validate_certs": true} 2019-12-21T15:39:05.262 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af', u'name': u'jlopez'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrRvNeBa+/GXxkSDmaim7U1wFpdBvgcK9/eP1I4fPS5xAWyxXI0s0ryX4Jwb7auhZdcqPjgUPQvIVVRridKJPz7l6jUlbIqdnZu4VHuaGG/DfIVxsSmw5aI5qI9Fg/s16JPwOUuzcLHsSSh8tFxY9L1yvs3ztQnKHW6nN5+zhPbW8UjmTNUksXXAeCM/OZn+IOOzzvfh0vVe/mghIW2JpOjHkZv4qwaWYcHg6+wWE4m8l84VIegDBiONknZXcQaRj82nHtfdnb2FSiV4+gKP18i90ai0iTKKXUkMJNFKp1T5MuF7UuJLA7l10FmgQJsZBS2Loa5oeqbHYeztNAfUxF JCLO@oh-mbp-jcl", "key_options": null, "keyfile": "/home/jlopez/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlopez", "validate_certs": true} 2019-12-21T15:39:05.534 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb', u'name': u'haomaiwang'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDFkUz8Wr2RPiY8+xf2hL4qMT+jA4mo1gmpIF8q7uVR03vX5NhzQVxVnXiyLCFv6susSMxpBafa7aLutqigXhDGJWUbsJII5yTnFYVer0QmudEsLmoDyvI1HCNk1IVY6aIESE50N3Ndx+agWasIU8TGKqJdDMPyiUtfrvsrGWi/91ZLX6aJSrFqBJtTpxDoNe4XL6QRPNv0wb8GF0az5graXkzVe3GkaQS9lyw8m2ougDvW1SLvoeE+lhUUGhJGkkCZKv1f+QNLhUHyQ3XkXqh4MrP3pPUKIuQ7nKiWDjCP6Db+ZcejjTRC19D2gLjhrw3Z2J7nn/loijlFzV5dp+BL haomaiwang@gmail.com", "key_options": null, "keyfile": "/home/haomaiwang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "haomaiwang", "validate_certs": true} 2019-12-21T15:39:05.825 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608', u'name': u'jdillaman'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA296in+8Sr9slF8hrzAV3dNz0q94Q4FlFaMYw08II5PQm6HOpabGOGOwOud/PPeh3lMUHhs45N5WnbgdfQdzz9lpDoNBcqxeACdMicdECMHtYWf2GuFUdTAnPBFVl8wFW5JcziQYJePUcAUJRXGGfdMLMppAi8VEuhMw/z3jz89UN+zf51Y9eT3rgAVSCR+Zn47kite4OB2S1aVMEcZBE0fPnh60i+rpN26s80fh3wG2N9JrPBEvst6LbL0aEixrlRUtoFHZuWNCIjntVMANa+idpuWSxN2dzfcoxpGLF1TFzqx0moznp0PnDs7ZAFFS1/EkeSCgBflyhPV6csG3VNw== jdillaman@ceph1.example.com", "key_options": null, "keyfile": "/home/jdillaman/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jdillaman", "validate_certs": true} 2019-12-21T15:39:06.102 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3', u'name': u'kchai'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAuTvkaUtEaLbLcYA9Zw3MoHI5ZPudKf6CAbAZdhp4RMVujimR61TT6LdRF20g6F7pH7GrI/1WP2bu7a3sZyCZ1nCJCOhrTTS+2wwy7oGo/0yY4l+Bw6i7LGmJgqpO5dDxwADMlaHUwJ7Rf1Q+Pd7NmggkAVTmaN6qUGffz9hZsj7ibD/Z8/GFtQsCEMloTLtnz4NB6Gxe54UXEKz9pvjpXBhs0xWsJtDKaPFrbKRKswuH45XPuwJhGr249ZYtwQqcgXSWhOUcPTwf5Y6fj27aL2dx566DxQzqzNbcTcO96MguIy8Ufnc0kIOQQw9oHJZtg82zwM6+2mWr9J/K94Gv9w== kefu@save.node2", "key_options": null, "keyfile": "/home/kchai/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kchai", "validate_certs": true} 2019-12-21T15:39:06.387 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2', u'name': u'vumrao'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2019-12-21T15:39:06.679 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n', u'name': u'dfuller'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/r/Xf58szqJ3wLdTE+mRK1Z11f/+VABmx6804NO77ncdC5ZvD3Kx6WP/oje01iwWEVXNAgEOaXp6j+U10aPvBMaS30qf/IhUcrXwO1YcGzZ0pV6UZMxcTuJw6w1VXy/YtgW5BlIfSm78zabztES9BqCgjYYM2rml3L1teDF4WU1QiI97iW0R57yc0t/gqXGpxqSFQ+iNYAl0Q1MGUgYltLvPrhm9WI2WhR2/GivdVMEM5lViNlegxneAZRb9YZMCG2kmSP+9Zd2NR367OrlgzmgeY/9u+KGXQiyfaO330s+iksxVwwE3fDl+KnqXuHSJNccJV7iXzZKej2QAire4X dfuller@Douglass-MacBook-Pro.local", "key_options": null, "keyfile": "/home/dfuller/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dfuller", "validate_certs": true} 2019-12-21T15:39:06.959 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n', u'name': u'owasserm'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "owasserm", "ovpn": "owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDH/YT1mGZg50qDdngGQ4Ea+jfJGZ582J6pQbYkrvuGsyATMzNwk73sAbn7mgbpw7nkW/ol/vwahXHZ5heC056BFnqiZ7tjoDeotDx/1XxFtkbyIaVflUeD1vBww38v6IslLHcVhPilauPVZSfVzEHI7JB0a+9OwNrEH3+dt1B6vR+WRy2gHG/aEJffPMv4yv/0FgRWy3cEiUvGlLq0KVVTbsqqLQhw/u5k8bBIoibBtZLFAMTrjLYp9Dv8dxJROvmey6O2TuX6gCY1rLGTAArYULpXH73OLIqaksJKc3srgoJrTSZPO6bo3BQeLz4ODJ7uvTpvQIhacBpp/dq1ln7p owasserm@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCiqChxNdielMPBchs0u6bxWstqU9oeY2K9qf1PNwEIrwDYTrpDjsdoVnS750tOBjWY0wr9Gf9ATOJXi89151oao9GaUueQ1rTgmu7GtPlKsuBUVNMFzLEa0mqneqDA4AfknisFt/LnL8fLM4BddT25tmQ4oUGS7F17p4kXqwboK/DsJciCMWWc/5AT4NVwdnYFzIA6ta25/bB94kpktaZ6HDw7G8q/mK62Yza3EZfq8SP7p0sPL7oVsInQ9AutcVonmALgzPa9t1/MPGaXUIqARaWtwe4MSlNEIs4afLFyteBVeq+dgU01yeyjFivwVmuS+aPFTm6sjGGe8/gOsFDIJiVertOqi663m7q1WYGlVANfif7uL2+uJ9yrvOmrFh5s/CJMJBRG1SLfK1j/ivFwvPAQc/uMqEIRzrVZOsuNLPO4zHPWzmBAPUteb+2BqSdUFYMlBxb3lmhFFffroDqnOK1nbdAfmyKiosUizQgX2Gs2wjjtejuhS9dpYkNu+sBOujsxa2mZUouyWf4fXyWGVjthrCZaM+FyeSDpJrgwgRiSKNB8aRxB10ZNo9WZEy6eH2adp0Xf6v2KsKvryDndqBEkBI8zLbGrPVHGSEqiSBWc+NeAB06uSscJd0pHU3hDg0qW0EaxOXaial9qrBiwcFVKEr0g7LDURQ6KZsZy9w== owasserm@owasserm.redhat.com", "key_options": null, "keyfile": "/home/owasserm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "owasserm", "validate_certs": true} 2019-12-21T15:39:07.250 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n', u'name': u'vasu'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDVFy1sjGvvHwS7Q6E61ykH4dUGfxW9+a7Xo3JMpTBT33/E8bTrYF6ZHSdH+uxScisV9Vll222NJCLXyx2s9EOcv4L88vWDTZGCaUb3iA5dweLm7EYi1NXT/p+jWDA7MAF1FQGSAQafTHpNRxLcrVPgsLDy35weV4xj7ShEU+w7EBDb75jlJXqy0luJLKNBrLzz/wDV9Oe6y3APwQBhtO7vGrVzrWEF/LWLe7U9ad3DQjvLM7+H+QVpNy+nt7us73K8vHDk1KrPlCCjrcN+fSGPD/ILlxrDrDeqmmBCtPMa7PSSjCyAV/EW0fsmArcHrHXKeKWTKXG9sDL0KBqos1kB vasu@vasuSrv\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQConeaJ2kvUOMy+jeKvXZTw655L3KK0ZG0HONIySFrRNkGa/M/hYf8oMNVh9SEax7vLNme0kwk7d71EhgtMGNsvHtQOkshWxM9P0XPk4HgB/OL+2LQ/7M3SwD4vuXPbqvrQS63YMoxcHBue5Ol5hXmM+qSoZajpUrfB2woqbp7aKsCpoK4Cgj4XfLHfh0n239ae38DU6GoAcL0GzMsxMdQhWzBwOO+ZHfowohc1pfY0+xGIMW1OlMN1eO7kuK2gGS/YRVOnLw9uu/2NErIjBzCihNN0PYWur2JE+KmU1x6VMaky49cKmAlc5WUqZ+FpV3gvnb3cLjBClnCkdjIcLeRR vasukulkarni@ovpn-116-227.phx2.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKRFyOO7qitrq6Hk2NSl28h4laRTxA2zezy1wwjTqs6UL+oZjNRYQjeWD3dDNPZrUmc9JyI3j1EN11cCsKAp8YXz3ItB46M0BPeJ4TXCpg36vxs7/twSDMq/EoXA5sRBbUJFf+bthcNemwG1WsJoxRVzMezbWBTHvz5i4Hbh4sL3PBbac3FhT/3b5Q4tKdO6bXGXmNLAoALyVZZMfoSz9CWzvrO+g8Houb9iJiaXHp293y+ND9Wicl9wHPckO1VcocZ+ldByujgX1cz6m+pxmf6 2019-12-21T15:39:07.252 INFO:teuthology.task.ansible.out:AKY1OMJj5PjmpN0HrgpV2QLVA+ba9qBiDwi1g8sRHXkOETMZ2jq/2a/mmTtvrsF vasu@teuthology", "key_options": null, "keyfile": "/home/vasu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vasu", "validate_certs": true} 2019-12-21T15:39:07.535 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37', u'name': u'smohan'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDlOImwCV0AGu5L29ABaefV9Vqjv43WYJYFuLTxGQQdeQGcways6iNKGIX/nzf3UWkwgg25OVSpzB7zf6FjBx+HcQU+RJ7FLzeaz0s8ZxBXqWwtOYOvP9cXmAXA07hD5UAfxZKC1twBzL82bKJ2ZSmzfQmBjvpUOqRs7XwsQIClOW/ycY+jLgLRLFFDxzADsgnn6UEGlHeMETYocUbe89P3wbY5fHhqZmn6e5ncuAQMsiLRa9d+QhSdpD07Y/P65EzqRF4UOX+zpOc6gPlgHFir9pvf0bTbbBG/qKN9+KGqzZMRJGKenpHM6EMfLkf0QmrT4s4zd4ZnUQPq3HN3pul3 redhat@unused\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD9n4ege98BNmvgUxogdcpptMILFfEisAQaiLUvSKpTsoy2gF7C6xSJXgDaziT5dW4ad2El4xmKQ/e6/e14Vx5i9pGbJz4NvL36Xmb3ent2PtY18i6d3CbnmGI8J7YH4YS9QvR7FC/KuvwLR6AdJ+uQShGhqG8kWJMjbqrtaxu8l1hq8ymBXw3O1Ht+xrZ/r3dEkAIVF3bMCSUnZpEbSsNoeiydEQl0ZCw+meXYmSw15vYVtKpYscZxqADqhSJJxIEHZA4E88dq6aYdV2AZ/s4cJK+tBl/ckBnclI3VGHSGn6oaHoO/fCHtKiENplDuKoLGQPO6xJ2Q/apHSGD7McY3 redhat@network\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDn/XgNfvqytLHgfBUdQIhJiWBlj+JrXqGZYvAZZJIWs32Ugm1/lCQTssGZrEj3+Qv81jLfkP6VQ8gXSWJkzUK4csxRxkZKXqcYLDE1csdmAFDOfWUFJwhMsCiiCyBiW9oapioLpvd40nw8k9qM4QuzaUhVonoGXMDDL1/dCgkbIjiQxxTJPHDEJFjtzul+VuhlhHqE6S1syoJG+UKo3Q1tOBCs8rwaPXEfCXFsyv+ZmcEv1+mpM8QRjdr1D1hfUfwb4CHtaa2AE6B5wL+zoY6+jsrTwZSwCfeIdgEZ8e3V2iXP5F7G6eXotu5r7mGYJClHZFLSfbNPN5J+EbRORl9/ wbhack3r@dhcp35-24.lab.eng.blr.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHYkp1nuRSYSAl6me1xuoTU30DW2YpAR6EH1c2adUC9tNtTZbMnab11aitQLAtZb9SKsRgnoB5KpH+zJ0VNKCuJU227r9vZwcVUruSGwA6X7LOrSkK0gFqNRZkVoKhC9oGHSU0zegYBDxx8VLxSV3CE56b+KEosEo6iJe7TkXy3wXHK6am0uhQPEdSHtdUdD683CGIffdF1RFmAFnzTPn96lC9rLe1mIGO+jVQx4zIo3i33RFQLy++4DtQw6Lrgp26Xr85HmQekjgyiya0VkR9Ypy7fgF2ucBGegnOgZDGCCY4G2Ruq 2019-12-21T15:39:07.537 INFO:teuthology.task.ansible.out:iZyKz2TZaOecDwrLkbYM7EGp8up5uLFRZsV shmohan@shmohan.remote.csb", "key_options": null, "keyfile": "/home/smohan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smohan", "validate_certs": true} 2019-12-21T15:39:07.826 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114', u'name': u'abhi'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDZePxJmYerihzA0aQbMt7K7bctPEDcwPHiO8EVuJowp6zwhSmtDnYuFybKJsee4l8+rDKR6T5Ugdl5pLjSzQ+Ba00OSH7+qWC6shvVDYju5mIooJnc+kwx8RgapbGtDdl/yYskBb1KD5g/3+i/2aM+Igk+7nQR7Cny5+WnUEvHd7PHBbUXQSuu+mlUlnXCVMM8g8qlc1I3WQL5eQoIqGjSWRgZUIsn3P5oryYBzbuG9jqLwbaohJvZRJqkimQDe7zCFk6bB3z39pyOWUN+sldRvyL8FPyEmqog4IwpjcYyCuCnPhnb6fNq9yB65RUIde82Kkzx3LZrYOXTrdHgD/lr abhi@trusty", "key_options": null, "keyfile": "/home/abhi/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "abhi", "validate_certs": true} 2019-12-21T15:39:08.120 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611', u'name': u'cbodley'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDcsBe6CzgWh64iZAH8zbU6ra4yEiUBSsiUn0AHzyQurvjeRpL2Mut6B1WEi0kJG7BAM6b13JrhnD9BW9yIMIUIPpVTa2o4zaXP90+bvjGrgDeJbxv3T9/TGh13k+WWyampxm2lo90/+Paql/HObQyyIv7Wan0DOdIe2/Ue4ooWH0tu3kj+3unJTuw82oEDEn+9tGs0oleG2c8XN0gQyVl1DzR2Gqf9PCzXK7hInC3niY2huoSAZzVFEgsniKEWqTnVWYi49X2SsJhUyHCj+mqrTG3YnW30nha3EXOWIbQGblXVoTIOc+gNreqdGByOMICKSnQmPSG4DAl3c1Y4h4A71T3Q043IUv7v7S9tAEKq2h6oJ9I/a5z0u4kaL/+B7Cw7l4bitTdcZ0rcqSvOkKUNNgxujbucMSSc4Tsh5L0762xaziD9nQZl5bteb3VtsMAueqhu7L3sJy69+HGg7oF68t5TguErXfI2UNVGOn3eRj+zN/zSwLUXMoROlpM5yHXSzHvpsbgZiHBImM2CeG4qaPtqzFw7VH6NWJv4ypb0htOMWsbXmLshNTca1KXrrBcVozSJC8wKgEOxu4spk26G8x1IVTsi2ypyYX+irtAqZrrpIPdQ0Ss9NjKPhFNXAxJgLHj6TlH8UNh1gRfN/QTyzpFSe4EH6uOlnWuxcuM8ZQ== cbodley@redhat.com", "key_options": null, "keyfile": "/home/cbodley/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cbodley", "validate_certs": true} 2019-12-21T15:39:08.410 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195', u'name': u'mwatts'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrtHyRuOSEXAM6mwL4qoM+NM4hpIKhaRS+/Y6RDoFKszkXFw/jC9ZYh7q39rQpWQBKHVe4RyPN9yyGWydFwaR/BjZi14ippmYOqVG4f7b+k39NFbMJRcSxYC+NvkqxK7+zpsnIWC+RdJ+blZi+zCbaaAYcMDSvBZW19FsJSLrO7hh4R1cuzgV+Xrmwdc1/VH+HqPMu3tHGv7N0SfsklWI8Inr0aiL83QqqgKPcVxljSWZGYvvFyTQOrkNQeaLURJNER9WIIac9o57WOINw8kR+ojebcOUBsrxZH9j+gMjsJ752Rph03wdMUpFcnS6yfvI4Ial7x/sGms4FE+wL/vVF mdw@pika", "key_options": null, "keyfile": "/home/mwatts/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mwatts", "validate_certs": true} 2019-12-21T15:39:08.695 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6', u'name': u'prsrivas'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtVTFNHORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@redhat.com", "key_options" 2019-12-21T15:39:08.696 INFO:teuthology.task.ansible.out:: null, "keyfile": "/home/prsrivas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} 2019-12-21T15:39:08.988 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea', u'name': u'pdonnell'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQD3vdP9QHyNFUNPOdplezEbdYsQfcC91ncATI6Fl/xHifmUbcvjIRqZHkCY6FNWUpMeg9S3+wlHJtzjVqJJ/MlQML3EtwnzQkeWv8bjtuZ07OqULYIv0zEx63XUfSnrdFleJ8+KDSpn7krkYE44q85B88IL+ZBeq5eCXFJ1TtmsDcvsN8WqZNWMdUGqgH3Bb4D8+YKrEfGxoievyxZ1A1DnBHan2l8foTKYggRrnuSbTb59cFpxY3R7Arc5Cn9kDJT1nDVBA5SSGCfRMiNyBOUzQVYe/UjKTaCpngglXC3wYnjoGvGcpksHLQCxyuJPB6djC1tdF+3Jehdf8y8+2TBzWXiiPq2XwS58sER5GcLLLQv26pPL8hgu6CdI0sHxgcgQBgpjoLwUMHmv13CnN/AAcZY4RRDK2X5zd6wFSEhuuCoUqh/m7K219IxI6fVkLaApC851D/ua/pj9OMjiOaJ+D13cPrgfqxva4eK8FQZvqQm7aUK/TT216QNYmn7hkzNjMTkjv/L5yYQ5Ym92AHy44wHzLPrtDB5mk94vHOjG+KlAt9ofuLncMSAXWe+fOqxuIQ6VfbL2LUv0EGYhHe4XOVj98q5utsVQxQbXAZzNQaq9FWXs2A4xcVP7aWj0Fooi17LN/W0ng4Iijir5hyLr7k7M1G6resgiY7p83biNow== pdonnell@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQCpgLVt7Zphmf6nqD+9BC0N91jdmaY6ZpvrYWlzbhxXITQ6juIEtegnlZ+8ohdeEfHFBiyu1xOSQSM1X1ugvpPQE94UqQg7JVb0tuoUCNJHYnIuv8T4YgyP0/vEV48k2G0BBWlP+IJ5f1iDvZuLHWLR5JSgGqh5ELaaHr7Cbyb6JlImldb7SDIoPHMCZ+n6dLO8l5Zy7qWelw6aeA770xmPucBr6t/BVns2g77XgJz6zFd99yEaDy27i8jfzL7WqR9i93yVE9EsL7tuOjg6G9KUzxBF0bg9iVsJXypvcBuG0JwQ+SDKqYw/cN4iLy5qBa6Siqrfpc/LKU21VNmJ0e5eXNgKKbMBpBb8GVmkoLaSRcXw7LclWopjNDA+meUhDAwU7b97N7cqN8BYhBKQyiShXFHMKLuNo2fSuTRt/icfb9VisKNGBdvwgV0oOlrqbXMzZKmj9amb9b+3JDBTGB7ioCsj5x/Evf9srXRrQ9kikIr1aBbVXIWir/lasNWVydcmUYsNi8L7W0vXArLwp+GgOXJFeZUiD3z9LG4CouRxMSwoj3lgWeL2ZR8feH4D/SP/+QmBdU6+aG97au9qreeXi5X8QTR3LND0DwwEKczX+3s+Fa950Oieom/gXC57PVKVh90KVoAGw3HzJTlZrauOtLkYbTUAX6woOMhAtJ5oWuDDT4UrHQj37jRrrl2dETLwKwUJN2+2L07vnnOZ4ABcGWLsU6d3bXeUBlw3pdsYqPebDKUEpjL2kUOw+z 2019-12-21T15:39:08.989 INFO:teuthology.task.ansible.out:mfjEfdVx9BfrvExvDX+3MkxccCJ8FY7od1s5fGEFHsXnIiMnLO/HZb/JN+Aoe+/ExadIcYwJnWcCJyly2/532kGFUlzOliCwPxgJpjWCKQAv6LjWmxoPXiA4JI0bp6y7lMEjmV2Bp3mwphlsW/r8Ie5hg1UPh/ttHRp/VZgcV8e6nNmP/qc6k9b2DWTNFqtWqm1hXLOwq08HzUEeH9mTqvenyAbv2RFMOS18Y2K+4BKUzHwZujrz9qEh/kqtpFScEIXkOJ+HZz3mI6XfjWFIPS487yRGfUvW3+nSWhvHc8vJY21bFRKdjD84qCVlZ2BfOCGUZSKftVItEiqXMTRs3RZd5epfCVZbJuA0hG16jUseeHqqZqDtlbLkzkxtS6v3iGRnPh/Ke/94FHHCzvYY8cmMYAVaM6p/G3edWftbaEtN70Fo7VKxPrpPzuSRZ7Vxmnb+Zad95DxxAC/vtMy/vkBVIP6dyn4HGxqa1oECFMe4oXdhnxZQoPzvWxb8upcXeAeOMMbcIlMq5a+dCBPGFNKhNhrb+9uAEDHwjE7rZlMJVSvt9v1S4hll6jiYIap+7n4reSKrS45aO9U0xiKXJJDohh pdonnell@icewind\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQC8MdUJzIvTCn0fM+hv1HAO/Lwjy/C/V7stQRWGIxyrRHbniyX8iuaU3j7YNFglSQPJq+YRM9UnGauM9/0J9tHKLsMp+zRQYrtxU/ThiTUN8TRJY11mGUol4yLsqkTtm6oq5sl6o+LfuA4hzaYc3XLrFHRJuCUgbtPG71Q0C/y3U+itrDK4wXUxBa4YzZzcIMg+c88VXwoouPbiL0gW5PCXzMaNtKlOqBUKlE935MLn70wIEfN/EwAAbss0ga1moqeqmi9Nlvm20DNIVFmkAGtYoq6CHXtL/uehzai5Sueop8Vmd8+1U8gDTo6JpiagdD8ZNf0zAbRx587KAT5ZMnahJpO276j7RNa85ZFkNkEnBLqBHRkCkTCYq1GehYyRaR69uuKyczFiIoTfAQgPh7VlnpT9525b/4nFsBDOkWxxWAIJR61YiKemPV/n/e9SwhvVsMDb2B/RBriwR7Mi2SZ/lBHGB+z0OA+EE2rGZy4LFVOfUczqIdpltlNotXan4M4F361i+bX19KE+f2RLySn3+GiFv0CdLwslS/y4xysz+FWxw6B0Y/EkpiDU3CdWM61+THGGm/DVhfL/Pvca7k8PonHFYw/x5w1nBcADpkM9JF+m5eOnp+weTEzl58sS+3GnoAuL8YFJwMW1RmE50xM0o5E3SDc8XftUvaSZz8mhII3uwDDukbELfmkLbnIeipQKyjO+GEx0JBp4Itz82qGuao0Wi32U9wHzhSWnX4GQ0pKZnq9frSlyCNwHu/iMiKF8sA2bX3dvplFJ9nKWPefcU8bdl/upr5hYrd2GttjUSkWZa+UuF264yx3ztcN7jotRhEmmdEcHVzKQ26/NyRI4/9DQpDKY7zw4O2wFkXWg5L9kodXwWzgy4cyOr4dMD9LBVLOpHc0CAji5/+lfuYQbu1mwVdfOzCXDqByjt7zCbeN7KuUh403sJ8PTQVp642xA9i4AkPv5g+fqi5C0++1DbZP0lmLfKtTswHhoN+kAqZu2N4Mu6mNVNKyZgW5h1pZzNSGWYcZttyYmMCHhPketReL0LPJ+VUsqQCVohuta3aaSe44InA28GZsXzMj5B0el/84bnE7cN4b+rfGD25NT73JEt3RgPa9W9jqT1hwd/nf53BDKpp7DlHjR77WSWT4h4F7bhbM1RsGSqAfZjCUJT6Rq+IuM2XcrLDY8zIeArkqaZLXMzfGj6KJ7cQes05P5LYfY4osXiq68Gs/7gzF31PQRRGOYAoqMN7752sn0ZMcpnUj3mJutQcncRUN4+naTMjU36WWD7it0YuQBnb2IJ4iKjYLInw8pr2zKgWtgr1TSCM4xKp/v1/ZMmbB 2019-12-21T15:39:08.991 INFO:teuthology.task.ansible.out:mS+dTzNC7Mpkt8vdCNjzZ5KZ5 pdonnell@aglarond", "key_options": null, "keyfile": "/home/pdonnell/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdonnell", "validate_certs": true} 2019-12-21T15:39:09.279 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868', u'name': u'jlayton'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDKzhyWb3PmU88m8BbgpEuBzYM6+x28z6aZwHSesiyfXMfF4Y7IZjtKaTxH0ph/O9GQCPhRCVhRd8TlgKWJGhts1E+VkQfHfT18CgmrPWzj3gifc2FJhL97vB1Pd3jksGnOPAikgYPHP46SVMTtyvXCkOUuo7E29WrkdpjawcqM1rfpw63OnzHf7Ar7Sqh/othB7ajyVQ92t+XWSEqj7K+NRfTaq5L5RBWbwarM885o8xdI5mimOZ0QjzKOhFDORrI+HEVlkutyB8E20nZgo7TUy+kYeLVhEqb3A6Pi2QbA9GFLT69kA/H9nebhj7Cuim9yrx4OKeZuIncG0UxSQF3QmWflAZHBs8L/jniTz4DJ5id5jR29tvGK2LeddCEMB1L2cHwWHlG0aX5jaNEJe+nG6H5Tcg3+M1RPM8twBkRQmS3UzAyX/MTH1kktyFUHn8CdBXe0WTuoFSvXLB6BCHWzDYZOt0tIV/xMtBwhy+XGYA4oZSsDU5m7VlSVLBGL8DbRX69Z8AGey2y3FoFu/roY6NmDhTxUedJ2ZfoKSwdZD6jjatdO1621nonBnJxrDgaXCWzT7HtnWcc3GwxZoBrqkvgpsx88f4lNi22OpYVPV6A4hECe04u6OJnHSWVk3ft3p5PaLNF8aEXbiIgzZKAwX3LxpvEDFH/t1sdihhpmjw== jlayton@synchrony.poochiereds.net", "key_options": null, "keyfile": "/home/jlayton/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlayton", "validate_certs": true} 2019-12-21T15:39:09.564 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f', u'name': u'rzarzynski'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCijPApnekNl/dsY+fleKYstEzNkKflBEcCoKtjItUCDJCVLVFOsDBrV94EGvDWpr13aj9sSwpUzRbeu/vOW8ZT853HH6Dqg2+eoYwG6vDRprZb93nw6l4vtU59/M0yKkyi9CPp2+YUo31cCFzYWpq8bv2j1xBruDSUCUdn9nJrHiLFqA4QEwSpw+uVCjSKzQL+LjlKmE8Plfd+8mNCQeY/OFsjxAztwN+4M50SZCnOVCIwEXoYRNd82JOXMOaqZJazSWTO8Tml6+Nu0hyEcvHkHxWRDSM+6LmXNnjlVUPyDN2h1/VAYATpOZ2VhmRarDoftCp9Wzi7eQdL/CGiMyuD rzarzyns@redhat.com", "key_options": null, "keyfile": "/home/rzarzynski/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rzarzynski", "validate_certs": true} 2019-12-21T15:39:09.848 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e', u'name': u'rdias'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdGthG4qx90VQ8mJ7TI/IFFzptkSqBHYZsFIVu06IQgHbnFBvjao6heGt4UlkWJkegPCDWpKVAe7wFuXDpPlj/VssRHB5LBnYHKGfkNAzMNiMjpEZsTdy+9w39CCujipz0AlOSfrUdANBZ11AnG9VgQSPEukX5487juob55lFM6J3x+v4opBF3rSlux3+thbr8V0w4oCHMD9/aHvVjFoBnxo4oirR4jc/jgC2fQCMiAJhFKJGhjkKFiNHxcB3npBuRnpy+59BcScOAzSXdJ7DGnwxQLjymnhqsAcWinjjJhOjaq6NaoeKv5I7zQLaVGaQdIyuuAK1Dnb7trpfQU7mR rdias@rdias-suse-laptop", "key_options": null, "keyfile": "/home/rdias/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rdias", "validate_certs": true} 2019-12-21T15:39:10.150 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953', u'name': u'vshankar'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEEjb7yrcY6I6XbMU0hCps1NqBuYmA6HDmMcS/l3i5wJRBitIXYVlv6pefUG9gdBYR5NneccLfVFURu+DC63MJa2ELJvVXsQ6YUOX39bJUT9KlZs8mH+nlTD7tlPmP17Ae3k8sT0H42Q+/5+b5YvdAU+03aPhzntmy38SyvLKrvowRk63wHgcE9p23LY9tY3+HSJu86cb/IdePk1GLjnrMeT9aM+5j2eETrmBjMArP+/2UIsT5eb8MBkQzJ299SCC2gfV/qIVkorFl9xdejctAwNwdT0SPTMxCHxaiK3wZ0e/5HHTbSgR5YlxOlKYq93XBVYIaFKPCePk1At1a3Pjh cephalopod@octopus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDcrTZoKO5e5xs2PB2UuAruvGlJESTHNAtqipbC8yJtRg8WGVDLt//rGu36QNSbpbJYMZgqgKsXBYolJB1ckqLYLv5j8RAoVwbW4QqKaV95y7x0bbXs/YAPnRBhEO1nfpT8GHQusUZvTbz38FDaQwstsYoncmsyZwBrqus9ycFavpY4wfpKoWtzeiK5Wjq7ervThOs98ZJKXJXOOxeI7mDakE3Jv7q2tBNu/rU1kzxIl65kWURzgxox6EmIZXQK3FzKt0jjdKtApVYSlx4ODLPI4iyxOv+MbRe3Nnj6FDhfz8/FeP/U5wWxdYx2WslzZCOI1BJ0VDLu3xEpLlu4vPzp vshankar@h3ckers-pride", "key_options": null, "keyfile": "/home/vshankar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vshankar", "validate_certs": true} 2019-12-21T15:39:10.435 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58', u'name': u'akupczyk'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+CtnvyVRSkIvD15O/q7/vQmD4oE8q7Mo6oT3kqTXaEaQ7iDayC2biUC0PC4OztPlNJxPUcKK5DfWw4F79OKeheqUbZRhsCC3Ge2JGoXHmsf0kBYdioMGiAxm+f8M/v1KESSSxvEnU7o+oi6VtFchh4Dl/WZ54rqZc/oQrlGNTHulFAuyoIAlKkCe3N0GdHN54PB+26QbLXvOuqpWmjVFbcjgRC62mkZl3LuCBhviXS4Rl5/ZE0QIHubCwV5XVrsqvscLS64bi06W8RN2vL56Vn3N9TGV4d2dlbE1PaBBESoTQ7Q0iUN32OAZLo7nF/1fiPY3Xc8r35juya5k+KR9z adam@TP50", "key_options": null, "keyfile": "/home/akupczyk/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akupczyk", "validate_certs": true} 2019-12-21T15:39:10.724 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e', u'name': u'nojha'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDeUJcLeyfqIk3ErPoFqh9kOnvUPsk/2hfaYNC2wZLMOSTIs3zXM8xjki8eN6vle0dmmeeAdwF+e2hJ3tCFKsEUQRvMc7ugIBA7L8/DboShYu7ItyQ3QRvy9pi6jnA/tlHA1LHalvyhMw0yZ4kvPbrfNMhZzOKzvL6jX166rqLA5AuiIfdYMBpE5rYq11GVjm0uQKw4UJpBMooNJeovEfQlNBTSV+IJSTiGBpBD7SgsuW3USMeEF3fEgQOcnvYdv5Fbp/crloyLWYc0HTidkLxpjP7vqY7E4BuBJug0zIB7voR3MYq6rAeuWjByPvq6coDlUjnw/QlrvarCbjjSaHdaRXLj/Dj+QqJFviESBr64ZynhHL21Bl6rexW7AwYyPa73Erbus3dN0aVQQ/ApcDTGHT7KRlDQ2YYg05YM4swpVVnrIdV58FzLWkll+ypCK/+kdtk5eRSe/KzZA3s7z4d4WLgolFWs8Fky8Rx1xDGewvMJgFUWVUbcGC13RocbNSxXG80cbIx8L/sclX4lDfGouu2sUXhP19vq1rJ7gZa6XTPohXjPbFS+En2cdlGdKXhppdvjT4QWKNzgQoF7qgWHMqJnMFJ6beT/2zP34KoM3t0ulee04z88VSn8Th8aqvLlGbdrt5LrE90escSaIAR1vNKZ1hCFwGieSKoRtYJTCw== nojha@redhat.com", "key_options": null, "keyfile": "/home/nojha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nojha", "validate_certs": true} 2019-12-21T15:39:11.014 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43', u'name': u'ifed01'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+c7zV5QmZb0WsgA2UZogoDNaoHcngRVLA8+BYk/U10ZlvCckJSfIXWMlBnqguZgdm/jBwsw5mGVBgHxUSqFNybwIlU00kyPZU6VbyVP+ZNVUf0XIE1NlemldgPBxxcOzSHkHhWrUqA3IKiCllcQVDQzutfUnq+jDT6NkVn4/cyNzNlargt7FR2L8gvskh62jLTBbhEuHMBp3jLCvaHJOk708gXm9qZ5iLOGV0pu7xdg58KE0p8ILClQBTbvoqPgZE9LA3qaOk1NR1n22kuW8JRHaKfD2Hn0Yu/MfKct783Bxb3mxWBKXxmM4otdmsR/R14NH/hIPce9wSkOOnQAC9 ifed@37526", "key_options": null, "keyfile": "/home/ifed01/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ifed01", "validate_certs": true} 2019-12-21T15:39:11.297 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4', u'name': u'myoungwon'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEudQMGBPqfZBV8bXpdQOgVTDP6GFsr5oL4gdbHFMuJCGe0Sfq8LTaSEBQx0MH7tXc7JJ1fU51U7DSt0kfy/PFeV1Y5XnNATiZu/SzdO3jHe06/8CCr3DLnlEmzXASq+tqDJbY1eV011d/TX1jiC8Qd1VNbpdZYExEDsDzwJXnWdh6rjTL4anGjIgQWcmU0XdbUJXvRP4dGjVvZVHZnflAkxLPBbAx/8BBOuBmv6vP4hNs4poi4VLwR6S6Ha1fgMJmsn6StsCVuOvKBJFuGvaulHgHgdoDJ1ryUFikkXXCWoNx4IbwtpQIkSxWMuAovlwwk9G8ozSs+RrFzCavE0f omw@omw-dev", "key_options": null, "keyfile": "/home/myoungwon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "myoungwon", "validate_certs": true} 2019-12-21T15:39:11.587 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2', u'name': u'gabrioux'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQDZPRW12u7LvaiynB87ad5h/JFtYIVjkKIOym9Y5CAbvatHhZ9fB1WAnYIn2HqiSSPUXT8sFy8w0TmM+aH0/ZGcfXoe0CdTWLtUbXKNlPI5h7TO9rHXESP2iRBpnKs3vK8WnMY0/D5BeESrPfwVScucmzaDrjUqjdZKR/aEeqVvDpQsdaCaoqNg23t4nhQ54BEvK8O+14nQ26Rtmm40QHO1ze+0UK+7nUMMFwxvd1EKjVp21EgYAFEbyQ5Kbbl+fmS/11AGy/yTai0c4qd7ohBzZvx8+sY+c1novG9Q7LhtyxuuYwzYIrGBCkKH5+iC7nt2rGBP+mnTfIgdbJUiXX3cdrMESwOSZy6x6zgbXXME+yfM0myhomKmUkpeZAQInX2KYrnEuPD9kteyK9tZcG6gJSaiUdvlab49Alx5lM2hc62M4s4WKaIgroCoApd8FtNAlTQsdYuycntgNrbPy0LXQ5hw8K8hkvc6RbxISdj/RXWguQSVbJWdS9Pf5NwD3Tq7dsOghJWhx4vBkhW3IECVrPQWVyAmgQ9oj1PoL7/2LZ6h9P8F0KC1qmhm+pWdEPLugOlU6lzQX7RJcukYlsINThM1RGwbPE4h9XcMyHAnODiDF63TDfQ01oKbVXj9XpOGY5eP0EQxKJYjCfWW0Ot6nK2ZCt8OYvoRkWMGnLv11Nub3NNOgh5TB/4nDvEU1cIb/Ikb4BRzVIEbast/pqUJmXe+GQ+ho7mp8y8a+lIxu54BR+VN6isjwODCVt1DYohgixEoul7p5pXvTM+JUY2GD/ddTGwPvVAN66sWnT9bjZZunQwTHqasFCIHNyPhnEnYIIqCo9O1I/D8zRQG3SYNxT6ED9GPARrE/H0TvdOmKaT3e0F2GRJRSToNxol9dpJovUziojOhXyBUjlTqnMY2RVilX90MnzgqR3zGO0aSruTGyNT2Faymqxn6T/50mSdJ8EyWH23BaaXUUW/z3Cjov+yCQjLY48SOWwlxKXw3hGz8xTAxauBYI01ojsVLCpCUGFyKlWS5iFwiLFtr1KPYqeCoS6CoWjiOiFgGe+w3lPPfp2h4rc6auIx2DZ1ByYhWLeHnpkV7sG+q5ckBIRAYmiKyXSGdV4zbXITMWDm+3zu4oj5VPRCfjMlout8OZ4TQm5PUdKZZhSE1jULG9tPDFL7nWwM3z0HET+d9kQYcEO6QB+ma9gZHp1qZJbXvCIC5fQ9y9El++A6N3dBr+ED10Ye88PLoWPeumyAmc24BDdI0skGk89Q0IzoLqagK6HGhPAso8si9wr84CEi4ncpizVLDU2cFKCpef4Sa03HoMbwfTgERtyptKFRbMT+SpuZsnrhF635h9uqFnszowUXV guits@Elisheba", "key_options": null, "keyfile": "/home/gabrioux/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user 2019-12-21T15:39:11.588 INFO:teuthology.task.ansible.out:": "gabrioux", "validate_certs": true} 2019-12-21T15:39:11.877 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f', u'name': u'leseb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpDFqu0Zg94mCjrA0FTB0vMiSBml1pqYCDKHhKBrfaza8u3bhtsmls9i+6U/vIwCEVk4GGPNhWduTJK0h9R5MS2eCf3GJcE0XHqRp/8msUCDPOVSZwC7hoYHZKunUIUaIUajOnR+L2slA/a9JaqataFH7zdrKbPdSKBzf5CXYrc0fK+Fz5OOsaI9KITbM0nESVpNCDzRqnA/aarF9MlN79vpTtvwQ+i0z2Km3cxqwV70GPMuFVo4CGf8pCGpbMrIg9OtoSC1svQ2vZzLkH2WrbDqTfYlOEYJJXaiBKcGhGnLZXfLIGSgvHjpv6xHo63yaMihocyg2MwTS8TZ/VtAaH seb@redhat.com", "key_options": null, "keyfile": "/home/leseb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "leseb", "validate_certs": true} 2019-12-21T15:39:12.155 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca', u'name': u'hchen'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzjJ6nlPn/OXP3knQ6orVId2QLswql9gRetKYlicKDBrdFg0MES03EIuAonu0aUR4NEReR7P1iB8eNnezrHyAu5mxIJlZp0rz7ljcBcTLL+11IJU9I9OjBFwzXGqJ1KK4B4Qfw35oc2O6F5V0JP2nHhHgyny37udJLImhxHy3RNmDwc86JhEBCVR3ouYiZuxgxxbA9BXc6+Lkdu9hEMMCDlaEXwNmMyASVU9Wf3dKnMTSCdar2PANquWbpNQZ5cgoTm39xgust5CEyYJ53aOdrti0DYK7qyzxUXre6ckZGrH4oGv2XrDaxpACIiHXt+slvow51gsZnBRHeayMAxQzGQ== hchen@hchen.csb", "key_options": null, "keyfile": "/home/hchen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "hchen", "validate_certs": true} 2019-12-21T15:39:12.450 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n', u'name': u'jcollin'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDCqZKc+/7iPLhapA/7SYmLxwyKEM95D6eFfsDBNKbEUey/B1OlqYSo+yKB/LdFDyUcd7shMDnhFFg47lNMANrsZdtE4iMPVAa1YhcrTgNbV3HSp9cthKixvU/gkt9Fnb6PEckRzr19wEwxtoNg3Fh/R6YCiSjPwqHmArtC1G2hzk560EJpozx6xylMR5IafatNDoZVw2YQ35fZvHbhHD6meQELducHJ1bKEEHxVIRSTcmXDpBz/vAvBoQOEPuH6PmC/oSwL9HVVIsazf65F4QfK98d+8NCK9TemgXyGDk3Sjz6xdm76yBhFzDQEAzO12U1YtS7Q7zGVsjmSDav2dd7 jcollin@earth\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC89dHSsgjgtIjJ3e/pcMthliQlX2vtdXL7pk62nD64230Fm1PfOVCnzn4LqDhBbGgJT+BrlwOPfzStpLgnUT3l67S834SJzhvSTTpQp2pznnQg1rZOiXjvNhbQwVYV4rphXWqay8weWvFIb5QfXE66mYA1etZtcOb1SMY2ofAjUiA8bAQS5aHpzcMVs4N1A9x5GJx/7/Bj1h8zPpSXikDYasJc3++UMGc45C/AAf725YwijK7xMnZNXv8uTBoGzO/Ko24R3VdvtfMVn4Hi4/FwuRd5bRYcJXR/nUdtA3IinqvlnTpFV2LbfgVTHiUMGhvewRTB+8S8ij4nnAlko3H3bhqZZTk/y7d4K/IerY0mozVIc2x8rMLfqhF9OKv1QFB9fskRZcF/ENHITnmLqCJPg9rgmyv1gCQ5Tbds1OI+IYQIowdiwtaViiTSIy7/CedbHqMz6aDnPsJFMs24lcUXe2d8uZ26RqMinMFGXg0zF7tifuVmxj7zG8DT7MFORDm5f/Hhkng05R5XZesStTBxpEU0MGuzWhENt3RuJdstqmY+wkDVW3XwjUvKG/gVYuJYfnBAsRZIZezw2wfvodhc8EmeHELvY1YtB3DGTBXKE1lsKGN7yV5+nl4UtR2IGzBn9GovWVi5ooB8UlSAeUmZix/eAA5teL8ldFrDht7+AQ== jcollin@teuthology\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeApVHu5er+R9h52RLSVYVhXFQSn/nMN38uWoDYPw2bqYBLNEAAMcPl+rnqHubELpXTGXjJYL4n2mjic+2rOLZpDgPLmOOm4PwgYstJjE1KLegKqVzXIa9HfASKMXaHSgwtC6P5vDQkm2MNl8vXuTAITF7VYdDS 2019-12-21T15:39:12.451 INFO:teuthology.task.ansible.out:33Kkke6ViJRZI7szYjbP7z9A1fy26THrvnuBT1Q18yy9vgk/sKo8vjLQfHSLvO90CJzouNEKqmkVpmJEqCPbE+s6Qr4+d2KY7L6Q6GAXxrqXK1jSDrrPsJguE6zZ1oWy/7XFMOr9ssduFWocnjQjPtQj3M+V+7FYAsf5uqQnzeMQTjO6AXYxTI7 jcollin@stratocaster\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDk/DYRp5E8Dla/4XHCF504X20W0u7CijZSCFWT1/5082NgnYeR90Prth3p+4xsw3YC9H5pEAS3XwcbdUb/NtWvZVIAIcMd7NOOEClSIj2Agk7NJxZ/4SfQRLrK3H2Qli/MVnrTxnEa9oeS4zqzv9dKRwFJDUFCoH2CpoumFVAp02uyzk7D/+XqFrXzMJUiSHXfl3YM2rt/vNmCtHtVp04UcG0Kyl2TXY37frRlSOkMMu0zbIC551gOyae6NGYhkYHa7u2SnngXcKM78b8MDvMv3QtuUN5P4WpH5qFRdlKNKsQGrzcQpBDY8AKQ+rX0K1ENhBwer49bihYNMKvjYmvTdLTNWgPCu7dt8JSgJQ8JMMhve0pjvFdpcKDLp0fvXDvJvZdLJM1ID/66kywJspHjShuqQNXRMfd1R9C4F20c+R9LY6eTGwrpLG1DNEez1VBHmOu5hQl03T2PfaThfLfMHRTw8JpGXGLMQarCCTm1PC6Z2Iw7KzorxP1eofGA5q8XoCbwGD01uQRZrwxPq0SHzTAkW2C2M8oEz+k4Mrx7+vGYzSS14hqxrzk91wuYhZCAKuGpqwm1ouJ4wxvWFewjbxLgucMfGSp09zJ2XSYRmtm7SAUifFob6Z9czHGTQerHCOTPuYqzlap2gCCVYtEvQyNB+4IqK/9FtEXTr28x0Q== jcollin@stratocaster", "key_options": null, "keyfile": "/home/jcollin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jcollin", "validate_certs": true} 2019-12-21T15:39:12.734 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc', u'name': u'xxg'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdxi9xdXNBYctMHH1cuuyNcPC4I4xVAeD3HZsKH+QwIILiKOTHu/6HZg7Yo1WsuwYSA0MDc+5LCSGu7RsmOEVpcSQ4gtNoFVwsThDyI6yhsm9WQqqXwSPs7ZE+urqd/BDfQ/7oMqJm5bViQMP0vmBExHuIFagj+8f6nspkE2Z7lBbaw8bEbAJZFx74oReOAELsL+Pbtf9Iaqrap4Z3WysmhfP6Rm/q7mncBBmo/fl/bejhThr8fghTPJb81M+XLhMk/rAdVrGdKV9iSt6lLiWdvf0lpTZO4LdIfJSBXr1Xv+lHRtaGXID2ahH7zl/W6dM4baz+hU7ZrbP+vdKg/SVH xxg@xxg", "key_options": null, "keyfile": "/home/xxg/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xxg", "validate_certs": true} 2019-12-21T15:39:13.019 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53', u'name': u'pcuzner'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsA63hT3ZAwQ9hsvfY+o2Gy15M1GRnRw1ZBSXEiCs9OJku7T9+LCvvH85Y/7V+mOGcsCZKqVTd5Vy3sHAvAF3M7CYJ+5IIDJKCYslC0wkKcX4jj26aMToAZPms8uvs527BNgKOcvwgIfPjs4WJG06fey5AhsuuL30etbAvbBudxRAKdZSTOsMveneKtXQu3Dx7y0rjfI/RBAhCAvisnMqYFUdajoJIl5gVXl6+8mauUOP5O0mvyvJH2jQveqZXTdsIPyp5mq1nA5psqvivWGz2bEJ+owI0GjfuYbyAi6i+BL4Bsd3GeoAs1G3Dicfr228VIUsT+R3xLBQUj3xIsVAZ pcuzner@redhat.com", "key_options": null, "keyfile": "/home/pcuzner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pcuzner", "validate_certs": true} 2019-12-21T15:39:13.326 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2', u'name': u'mkogan'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaRX+aBLjb/h12ctmvoMvZro6NrrYDuxROF6QMVpEVucA7jHhwCC/j9dJAq0hOVBO8R3S0bvKQ6bmnCb/4y2J6tq18xKUFCrChNii5OQ+AgM5v1353oZ9SrHN8p9ajvS8w313o2OpXhpaPv3khVDjKZsDmeH8YL/Nx3WQKqIJC1mI4jrI0sZGGJLifEy3pPkO0W1KmguN9AwAWK6+kckAjMa3ElkfK/xuAqBYgNKQUxP9oMgTQRcgpWF8Nzb0npN3v+HOUIRrth1INSchkDKtWob2T+0Zqxwm+Vpi0iFWwqjcN/OVdaDGf34//A3M/tsiQ1g9SqhdhZJCfu5XLmM63 mkogan@mkP50.redhat.com", "key_options": null, "keyfile": "/home/mkogan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkogan", "validate_certs": true} 2019-12-21T15:39:13.619 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed', u'name': u'vumrao'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2019-12-21T15:39:13.910 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736', u'name': u'henrix'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDZDKRAePFqjugc2ljw4QmM6r7B2dPI39OBB+2HKONtDZI4q+uwgZ1yqnhrbJaU9ux0W5wNzJyp2VfOPhw/9G9jo+efJPg5IySnTonhi1zeCXNpBMY70dJZVCyGEDdWKehvFM9xFBOxBC/h7gOyKqEc28BKoy4cHfULzm4er8PU30MyFBngu8NaoKqyQ3GYAfWOAXF5lUdalbJwl3a+HBb8PjUhbGuznAZF1Ji4JgnNuEZdlNVOKwhEkAFZ1JwXvgLfrEJVQs5b2V9Sp2Vm5ZngsLg7b3RzV5I4B3Lj7QTTraV1zLJGiS5iRdjY7D4W5Uxi2k1X7vaOjySJWWadHyeicNNBhLSfHl6JJ3aMuA8oAAmtgo/8IkJ96GFtgM9j31Wg8wkeTyQrwSeqCuv1uDFr8rp+OvMClhHnDgUklJLWjjYPCDVYR5KV+iOiHL58WJg/b4h6NsoP91wO8P8EcSaiMdq7P77w5nJxDI0KKb//dC8aHtWnRRRlrIS+PNPqKJSNTaSwOTieXvzplhzKFzu2dr79qs+/67iaNE6HBvIX5jVHWCFV3wllyjsC8WYPh4kEVtg2SY00SDOOrMrqANV8DDtHFwwxQTEjz1z5ESuwy9UcUSQfP1/zsAp2OQzPutKs6UipHyBK3pLm6MjaeeaEWu8fmetCqALxYV3h17O+Yw== miguel@hermes.olymp", "key_options": null, "keyfile": "/home/henrix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "henrix", "validate_certs": true} 2019-12-21T15:39:14.199 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26', u'name': u'pbs1108'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNwmgnIsDDJmsmYzkTAEo3iLj7Jd5Ni2R5vD1Z0NRZ39zJMtbicZ0ec3nAMzFyHrmPnKNFltS1mhehbLbtKNKq1fSo2SMHs11z+N8Nl94F4AY8IhXZ8U9GAorEJFVIxHUNEDqwl55Ov6MfP60UI7rNgy2rUt8aOBmAJVhLEFmJJawJgyPhTrUZAT20OoILphrt81o5bTNfmx++Zs5Dc9VtgIh53hexwKgTvG6fxfKdra9m+J2GXpniOsZjPJq/A0je+t7H3UlxCSu0dRy2Un6l5ZAcoeg74SP50dFgGaxkTQqI6wle6NjtPIdssgrdYTgTcOg9M4jWqAr/5drTUSGh root@centos7-bspark", "key_options": null, "keyfile": "/home/pbs1108/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pbs1108", "validate_certs": true} 2019-12-21T15:39:14.479 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315', u'name': u'clacroix'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDnvVnbK2SNfL8hTmrA09L41RudKW/F7Ty0vvlHjrFjDkdWXVgrafFWGKQZ8dSR6FJ9JLU0X59xiEo4z31ini/Dt2XozFMIr7RdeB3xpiifuUUG5Lgsasd2ezniUoOEukL1a9KkCIxNgmKJuvtlHCt+ZTtyL513yjRICByZbnmBSSpUiett50X10vtvaNBu4LikUJKLc8gP0PHLbaGsU3uOUd655lkFWjWhgExlU8bHWiAZq18Ss63jDlSKFVSk903GKSNcHXMHNPYaXlrYw8NFzejoy/XjxEfmF/Lv3MeAUhHQWx25sh5aDv48mzo8FaIJsL1MrD0QxF1gllobWAfp clacroix@redhat.com", "key_options": null, "keyfile": "/home/clacroix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "clacroix", "validate_certs": true} 2019-12-21T15:39:14.759 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006', u'name': u'epuertat'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABJQAAAQEAyiGmj+nOAD+il+FvXvZSpUOJQp6hKh6UPaAIQvQZkdeNRWN7wcDXMUlLj+TBFWa1OO+9xqFlnbgVMDR4qy5orUBYlMapUb1rutjbk64XeY58b6uguHwSoMd81QpCCIN4KOk3cldV8UiyX382baoZ9Je2hsOnnBGMsRtHuVHZ90Bj1rIITkwajq//KRs7kjb/qhjk0xkipDIcaSwKjzPlmmnYn69noODwDexpvLYlCF+LowSZEdrOUvVIigSFG7cLTDaU2tLhxSgbvWu8f0pwvjsyW7gOgM+VeRZ6uHC79F1qvgdgQSxflddIgrQpH0rzf6RNxQjR3SBkZ4V1BESa0w== epuertat@epuertat", "key_options": null, "keyfile": "/home/epuertat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "epuertat", "validate_certs": true} 2019-12-21T15:39:15.039 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461', u'name': u'tdehler'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDRZyWiDO5JqWHJdUUBr8YReiKXxnL87juFAfkcmw2E9KmANVH2sn3K+OgUCQfBs1Wdg6AbcL2GyMm9alodUruzOm127/bH9OcGmu+8SNsMAYrAF9EzsKJa8Ba/I05rl5a87p1jbYG9Rpo7u3OvLx0LjgqydRYcmjwo/niifYNDTaN0K3v8s1mN4TorUZW/i3XBMBIBjlIiNGs/I7IZreR+1N4V8Nwrp2EWxDv7AwG3nvokL2RTHcsV0KgmAnciv8nXXjIf+uqOPuGA+F2qW/wz92bG8R0O4CjH+QZt4k6UoUIXFRW+LdOVAqmUrb4W0OJ8vsVeukW46SvPnYUTkZnF sonea@think", "key_options": null, "keyfile": "/home/tdehler/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tdehler", "validate_certs": true} 2019-12-21T15:39:15.322 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305', u'name': u'laura'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEBD7CxJJfbLW4jtuLnTwkxVbhCfh/JL3Qj8oeLEkny1tE7QCeXn1bP5bScWSgLH34Wh4TAuH+8/g/koFwcCy1YuoGBdBV4KV7O9MTnFngdqNY6LKzCipm27tnWZEa10hrF8dRFTmhracd6kmmVGHgvGQPVAVONmix+2IZ83wdj8XJSQHeE3PHFLO39tU+ez/T76QaiBLDVo4ctHVt+N9bBbHV6Cr8fyiUUU6YEKMlcedvxGpnMK8oeOSNuDeh+DldcM9KYzNGEOM34jGRIvJp57dxQDQCtPPW5hvBXg7yMyrv6TcJzDd2LFD6AC/Qw9yrEQPXXyBCDkAlCqDR0+rH", "key_options": null, "keyfile": "/home/laura/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "laura", "validate_certs": true} 2019-12-21T15:39:15.602 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce', u'name': u'yaarit'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2k8xsrsLxEqJx3rV6ZZwLH0OjYCDB2uIYzcAyrKEOKlVWO1ICKxkw+gZ9tSiPFNLfq13io9BIrAKJwlrdLCJLmlgCQwpwUQqzmRlRZEm+WmD0dl8LPeQSidyJkPkJgYBEshFPjDHIj10Ei1pJS5svI2l7G1wMWxlt2kHL8Og/4zdFTRctxwqZcvzLmRQcs8lSSpqTkhfeKfmrpR+je8XFbCJJ9m/nJbgyCiZKaGN8nf36pCQDEva4rvaSrK0a6YxL0RWN8W3lqtBPKIehxffr3lYDEkD9EUW5vf6AUKC0ioh4jkPo9WCjaoFsOauMOMVQPHehEAMufEO4Kym6nfrf yaarit@localhost.localdomain", "key_options": null, "keyfile": "/home/yaarit/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yaarit", "validate_certs": true} 2019-12-21T15:39:15.889 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3', u'name': u'rpavani1998'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDUznqo3VDmcKRL64P7sbjuxXCW/0ljFTc5zZoZ9UvtZvU2jrRQ2X9GyxcrcQnAWlF5YZHqeTDU5o8ixn5fq1MSmaPRBNDKKVQOfR/vFQV73LiKx4BFsC1/EdaPie/XqltaLDTtfqhxADJlz1H3a3ZlkeVHMROTqo/TidL1YUYlRH4qHxWWBuFuVS+EYfTegSZdjwp+DuJA0iN6X99+DfH2DhiHENjEmlbKN13+H+EPzm0K3ya2w8D6ANodB3AxMn2tGs8mLtG/P4kCjoDnoUVFzKTyUpTtgOjis47uSY4dL8mTbIK9kZAiFxnT5sMClu6KRStDjRIrxbTBQJ8wvlat admin1@rajula", "key_options": null, "keyfile": "/home/rpavani1998/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rpavani1998", "validate_certs": true} 2019-12-21T15:39:16.166 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4', u'name': u'rishabh'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4NYxUHx8HMgbVIHadruN1kAJS5be0aZA9rvGtYDfwD5siKNfXFRyLG3thkjxtEi7DPYMxgbpFOXW9EQKh5sQ0ohsADZOi8FSIAfQlOQ+7HAEp7DjBtjxoqB31jJ1e293YtO+nqAhcOMyGfDx+71w5sS2EFqy7PscA9gD9aV2iMxIR/JD1LxXabiqj12jU5AOxr7ZQ1+gM9aroQuDpNxY4U8jBTxx5OtIYRkeI/aN2gQj1mXGMKGo0ItFn4lsXGLxGxMpa9K2UFw2dWQVTw9VNm79nixrCX+lC2nyaJPsRYZipXF1ID2NRk7oJ79dJ/uRBBXpunihlEC3aF+nOgEPH ridave@redhat.com", "key_options": null, "keyfile": "/home/rishabh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rishabh", "validate_certs": true} 2019-12-21T15:39:16.448 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2', u'name': u'skrah'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCeAhoTum3/VmRbxSyw5gwgcPF1EWFK5mEPX3PZyo3xPsUAJazALfge3aEuCrZBbsjOCoj3GfFhr/ziEF+OWvhBoPK5hQzP0V8tE9cWJwvHmD72vGE36o1hFO0ONE4DEuu5KRDqQJUgeyQWXsP26uHTwIcvMpy8egZOfyshqF+lQVXzsAex7Ninmidsf/sRNLhzT1kLrKT+ZMSbZ7Rqh46TMlmLD1LzJZNcb6PGVwlfQZmV6T7m0COu/skeSxMcy8hqMvHFNvvfFSqkfMw0KBGLDxMQr4L+e7sXwQDnEZHdzoh0iS3xDp/Onr1FSmPyEjz51e1sMHBsqklHWidSZVLt/+giB4o6Wdgzc0um/QxW2Ne2qjtDlViOgRE9Fzcw/cRu/Xsn8qtXTxYyxXswjkA8qX7WR22OYgjYChw9siUsKkuD2yAbPMVl+cZWDeBhQnPMbDMVaghS4SAVt5GK16jij4+r3IE9vGHBmGwJYJ/OrtcxBrUrDopwsGLcU+WHdv0FXuiw+YbfnzpthC8rVj4oeJ2IUOQ5JB0ub7NmbR4QWjc33JfD+AeCB75x6io07V2riRjMD9gLC5nwytJNG7Yytf6xI2i/CprW7HhsDs2ykOVcDSBMnL1oGW6FeHaDRW39r6BXSt1jjZQ2E6pDEBTDO41tPEL9EVvv9TK87GTylQ==", "key_options": null, "keyfile": "/home/skrah/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "skrah", "validate_certs": true} 2019-12-21T15:39:16.736 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8', u'name': u'smanjara'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9/Io6M3q8yQqoPNPKAPJ6Fzq18DMfmNtxcD8GUWNo0xcMxd7XOlarvYc7INDQjkpzx6s+aqMme55pSUZuN5plW+nuFmRPqBaIxyeCnNjzCch0wbtEZe5dzT270FhzM1Tu4W0dw+WD6zrGjXayLNs2uxNp3UrWtMTBRzd0IfxJ1R2aJtiCE/HNxIZn+bhS/OmEPV1izz23MvKXD+nRaeJK6Aqrg1zbnpoNskz2fZ/AnmhnAMc4EHl/NUyW1mi4g81KcQMwE8poWmw1NpZje9gWFcISVClLvvhk/A6at/v90P7PisX+/ulTA/QYJYvGBjgbNtURYu/PWzYBOAEZQGfj shilpa@dhcp35-184.lab.eng.blr.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDqRSG3i8QY7t9sp6I8vDZ7LVdk+DDrMhI1QmvU2z+YQhUGDBOvvO6YP0gUXPDyalyj0sJXiwLWPmkrOsbnR96YCUcIt7Ba6R0d0w85JxcJ1WyPcsg2UyOBwq4A+s7VAFZm6FEWzlaubixsNp04gKDqBjSjaqA1Kwx8/Tg39byay8Hn3VJeZ1SkRukvhykz2e6gl7ydfuJQVmEyO3pTrd5157ZqgvQRABT10wc3vR2m4eC65hU/v0ci7HMY9oVtGEdWVBapWMuiocrOPCRWF5zrMJwW4vzPOCHa/47hBEIWfOWJidUVzAawnM3nOe7PykvvDK/iP8B6JpxKTE74eLeh shilpa@fedora.lab.eng.blr.redhat.com", "key_options": null, "keyfile": "/home/smanjara/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smanjara", "validate_certs": true} 2019-12-21T15:39:17.017 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41', u'name': u'bengland'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDpp2NWr5T6s+jzHiz/FfCKzN7ZBnUSzyE7ieuntzjeGotyupAAjVBoSpCJgNDybm+WkE4dShv5mttPJ2/q1UZCLRj3nN+y3NG333aLZuyugR0ht5qtvGVegg58PwQ8CMobR7ca7bl5UdEy0UIrM3d/qpDOuPkuE4MIJcrtZGICE8NAVnmHNOB9Sm4ovAYkewTuMAv40j2Dl4F9dD6bg+OJfBOJ9+Zj+MpowYlAILtEB9Dc20zB8GGUEtr9yfOVroblTYhvTxGYRaE7aVBe83DNwkWGbDc/qUNR6bpA04w/P+LclYVsVKRoHOs1EXHSwNMFromA8RsKS+rajwu5uRg5 ben@dhcp-185-182.bos.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC93wGgyyQmTxa7f+BZcYPUapTH2N1mnV1ikl9nzJzHhgrtJ6pv00LhtH1HqGhvoqnqZjBD5XRAKyjMRcMBT5MYcXXep5RqZ9sJ6Xz8rHkBjfl/OOQUeZ/ZyG/Ksr1jpGraN2xtrjQQzbqN1YvirCqfLk2exPvVeEk4CGnNybwKwCWob1uKV46ASrYeQK8pfHxrW7pTo+xdErHOvb7uumZm89/YisUJGicO1qSHBeuw1MxjhnD+fIcxHKWY5NNJ4qScW6a5AABk1JJsoHkh9IGCVaf/zJv8D3gil5SiM8LIBKaNa1fn6oObPl/Ky/wkiIo2gmUZawvAVA8iJnX9eUPl ben@bene-laptop", "key_options": null, "keyfile": "/home/bengland/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bengland", "validate_certs": true} 2019-12-21T15:39:17.295 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d', u'name': u'pnawracay'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC3YhI4am/T6s4rJRvbC/jAvrNcDhE/zaJiVDAjEuA1W/5Xt9ObX52eAnp+tOOtelXrX8ylYF/z5wGNQLjYrWepXhuqaEGxeVayCHzts6LMpmvyZ6sYnAw9HFO9rzrTawm70DuvWqY2Zt4KA6fM+lfXk3Qdpi5RCPDKk1jpXwSCyUtUKSWTDRtYtvvJTcO3H/xCJxQq/F9sU8yqgQcqFPfMNrV4a8gNj9KACqjt4IEIfuxKlKhlMA4yu1VTmGcNuhQS+geJzsEGuxZUQb0Io+q4o0b+40BQtazKgOdTcQbgNtqh3wZpjJsvB4AM+GNPjZY3mCsd+WPGg6dqfU08Voo2b7BfSwyCn14fQHnj1nSwRIea2juFY6AMfVJjstM5iIz7lOH2c3ydWWhzvSmMeoQ47zBXlg8JytWmDrXiyd21BSIpTL0Tn8P95wHYka8U9KRV0ZY7Jr2w/ThPdy2C+Cdir+P+OEL1sKHSnmEyQEeONa9P5yH1Rd1kM/FNESAIRiMLGspzXQeWoJhHswVaKoSBW/7f4b9VENxda8qAQi6ob8A71cAjbpRrkUmd5EBCSFMNs7lscus0/deUVvlpsrMjDAJa8S9gqhGHNU0YvF1WBIz3paLRAHDm9DWrHc5lreizJzX5ul4P/Z4ITjpO+9ZxKOcyi8wnJ8Og+QUYLpqOIw==", "key_options": null, "keyfile": "/home/pnawracay/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pnawracay", "validate_certs": true} 2019-12-21T15:39:17.585 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8', u'name': u'alfonsomthd'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC709gIxs1RgiA95ddp2uymzgV84VHJVVMj71FyTlz6pa0gtlaxF+r16p8dnNWjhyJRUXMwBKFOwHx95bcdfiRA+BfC04UnI8VvBxoED5Nt9EbKD5h78ZoNEeQJKoG1dsM/kWWlTd9WCsCyzjRz32+YNWul0LOhrfloHP+LHnfjf/uOnzd8iRHgnpWr5Z9uB0MBu4y/L6AmHnR+8pMZQq6Bfno8ZLwNCzx9v+4MqOz3jY7FYv7SJIftQH0fJygIfCz3bKyUEWk9a/JJacETuJGBnHgNAXUr/FtM1IFiMl2SsTixRgnosEz4pX8EIe5LJyw/Qudro8BPEC8aIGy0XAkerGmTt5iALMnzmeeanZCFm4rF/YiFoHLMS/B0uv42MD64hB3knrV8Bcq+mRjwVOYgecO+xoiBsU6FZeXaBk5IsjmYTwataGv3ILEmNpss0WeqHxF9nd7oTLnw51cZfcaerOnJ5IZJZTXwLNVr7K176BunHggAhjvtvwdeKhphqt9f3ilLr6gzksEoAR4JBOYJJtd8sEqHwBtti+YbMGwH9Y+Z+SujlQBQIsom60xj5OzrwjsIi5JAtOhbc1Mq2fptSXRsR8ZBT8aUFqZ8R7ixEqq2F47OAznbhSQKidzwUNkWNI3WAKi38Oi4aya1k3iqtL04bGf8R/xD5Vq/2J+UlQ== vant alfonsomthd", "key_options": null, "keyfile": "/home/alfonsomthd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "alfonsomthd", "validate_certs": true} 2019-12-21T15:39:17.866 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e', u'name': u'oliveiradan'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCoL9+aMxLtUG3aRIuT4kbjc1CPuQ+wbS30ocuJ2YIExlKMSxVr+LYqbjcz0T4f8gAKG539dZQgD9BAsHM8yfOVEvei0KsQHn87pzgYGR3DtaMx5IEG5xNyARPu+VuEoe8qlL3cAPwTiaU78datD5Cz9GEClhL1Fhn/Z8o7BMbs5xlcUIQ/J+s9/sKMcqmmBJy5IHQIEUHCl4SFwsev5g5ADlnLNNxjpalExuBCEeoSma9F3b2YEJHvpA5JhCIMZpVfXbgh2UhUsyfAQN6KgOIQ/XlKBMQDtizf2vZ4b8hW4ELcvKSADfTsyyqEs/Ql8jEJu32GFKf2mYPbgTS6E7FZfQz77X+5OBhojFAstB8LSSrH80FrG1gCbVZLnW0jXcSehKnU0oNmUWOKxxaT1b00fo/iZ4P8O1uJRY17/uaVCPNKdGTZzIlzyC16wsFFBxBl0d7HfojQY6qfww7AQJhvDkV2sP0/MIJzwriWOzZEmfr8Osqery5vFiYcSvMbdMZDfMEYd3z7Va2RiTIeyXuEYp5hzf8eyJRP/f2FEw8/BBrm5JCxeSWUx7qwgPHrJoWAVxCIg/M3YS/vfVSFc8Hh88sgc2upq0OqYr0uq6EFzUZi4aoOkOVYn9u/uc/p1AR0KbiQOkERMUDWTyADxrHaL0nQmGy+XyiJQjhq5ckZ+w== doliveira@suse.com", "key_options": null, "keyfile": "/home/oliveiradan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "oliveiradan", "validate_certs": true} 2019-12-21T15:39:18.157 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba', u'name': u'swagner'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDh9bzZJulXGES+l9Xh6Aq15RZ8uQCGuDNhlNQtDblE1ISKJ/DqGYXp6wUW54+oVNA7eZiXz+fi3mq5pPEtZZOfd3ixEDzDJ4E3cVXoDZCqeWEmea6KvybeY10YxvB56TEI8U2KKAd56PRl20klXpLCjzXNqG7n0aXFcpCMbXDu369VX6lOk24K/7++7Wc5SttcvVu19sT2kqzsB/S1Y5cxiE6RM6wVtqBoksp/kRCIA16ruNwx3GUabDfbEoUGXlkkUP7+TZgAbHtBsYy6mCQhwi0S2+WG+HhHDUPjHhV+MdN9ffibCtOEGo52itVLVky09VeBocuA6H22JDGXPBgjcgf2NsZIqcKqGHhUkXmH92fhRSFOBLKHstrBq8jWRP/mNrgj8cQksDsakQYQbDg5dyabp+M0/iL2Q3YVq7erZI8aZMA7ZF3WgoQNYZg5E7oejM8URIlFP3x1ne2ClRC9a74phSCxeU/NVamGN3G3dImzEXGOSNyRggHJ4jGrIGc7tLPCzmI5OkomcB5OxReqf0r1TNXuUAqw8M4EoWt+0xoAmH5zVlUHf+psUCJIEV/4pbgtoJiSNq+LVY4jyDEFbvTAL7MqyXorMV7Tqlj+/3d7RXfhW9lR/SHb1Z3jcHvz4ZzYMzKWeJEjz+Y0NwIdFDhcPmUOYtEDmuRRrgYiSQ== sebstian.wagner@it-novum.com", "key_options": null, "keyfile": "/home/swagner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "swagner", "validate_certs": true} 2019-12-21T15:39:18.438 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e', u'name': u'yuvalif'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDzYzqIDimPjMOUOUz1A122eV52fl3mgGgNBRYRlOWl/lOxM5oOat5FPkJlnXTVyQFl0MlnYgIv9fTnjHVTM0wFmFUItlUl151ZmK5XExiGpGVsqG3UWNyKJxZpk9Uh2a0T/5CKzzwXnv2fJMNxzfqgopYDrgq4TFEZcdvgWJ39dcRTXST/rGJQ8wOq9z7UNzKU5VOGxvgi6f/nGevGIHrSaTcY3Uu8BW8kXUrf9vfl+ZSxVjuNHXF/VmrJH5Ne8mVu7DzPMrl8ZZjxsfJKT/eQqF9kjqht5586OSPKIwg/LEOyYj7Wyyf3QbDAIp5bCygQfmBBl+kIxphJYq3EwUZT ylifshit@dhcp-4-102.tlv.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFgwQYD9amVSYq6JzjHyGmKBJATQCM9xsf6QnF3yb0Lvry2nzY9F61evVXQ95TmF/AqasRx5p2WansIvegG5ag5mytvRwFXtHzPs4o/J+XjA9o4+A5l6j9cOFLIDD/TwrwlUjgMa2PhqUztSpI1op906391RLrJAyflfDghJxeVl6dJab6TEsslV0W+yaCS3BBxMXXrCbxCB5Ms3KCRKDz0DqbhFc+Nw05L1O1pC98HBFRIp1SaXlMcT3++CRQ3wugpOMkDMNLbycl+vdMBFY3UZ28rRowS3Ta7V5EARZVVH7KxU5cL+8Hr/8DE1/ITOmrluaEZ1KwYgoxZbZ3w7Ptw0FK6aiCZ49YtqKT1WakoVfO8m7pCdxkqUOA91loMQtxvCwZ/Vfoxaut16gklmPz8N11xvo1H5XB8/ZrvK5yKdl3JDl+wwtfE/fwo4E/fte+rPRisoWBRsmsx8EGxs75sXnb6n3VjZ65vYEKsNMLg+5EAaevWLLeqF0a42UWGJrHdb4VKX/OAplHrOk7kA/VwArow8f9n+iaRp2hgDU9OhQiFVRhgqwahlmCFFXkcP4/4WlxMCuZrsaNMZgMjVhfsp7cskKOezSfZZey2MB2fREMS1wg0j40CdYRn4jTNEBr/LVszsoM7t7AwXUpnqXeYIe+gePx1PJb5XSzSPZ9Pw== yuvalif@yahoo.com", "key_options": null, "keyfile": "/home/yuvalif/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuvalif", "validate_certs": true} 2019-12-21T15:39:18.740 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852', u'name': u'kkeithle'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDXa9uTf3q+7tGnHzSxv0WbgpRKrBIYjAZ5x3V74vQUSD1aAbmyvOqgPnts7XttLYImjjRqiisyC03L3KgR4b3SOeyL3KFxrk3KkHAJ/AJH+Y/UafyDhHNl3gRNg0B9rjsa7h2dSlcrFKnAX0pDY1HapuNPrzYWrXsEbo7C6wteFh7PyPmTkHu0NzIIPC4FcLl8SGaww/HeBh8v91IymgFSgXG0jNC063FEExWdti54yNLGh3gpW0FytsiRN9eEOKUOgcveT1tejz9ORv8evO4Y9SCQ/lsyOKQKNcG5C7J8nsbwSjqQ2cf6i2aYFZ0Oq1UXuhJ8APJCoiKgLJy7AiYf kkeithle@dhcp-100-19-89.bos.redhat.com", "key_options": null, "keyfile": "/home/kkeithle/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kkeithle", "validate_certs": true} 2019-12-21T15:39:19.021 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130', u'name': u'emmericp'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIBYa5uEOlFv1s9K9D1SbcaApLwX3LBwASSnRXDHI7MtU paul.emmerich@croit.io", "key_options": null, "keyfile": "/home/emmericp/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "emmericp", "validate_certs": true} 2019-12-21T15:39:19.315 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0', u'name': u'mchangir'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCj28V9sMybkAzhkiaNiDDr8TfNVyr5DNdbOqszdGEVLy1Rxh/UYnxUclsR9n8lt9VnCft3M4NDTnVwlxOHE8+0oolJE9HP+DQomsgj3abP04EGJyOA0X8E7eIFQNLRbMFCD0yX/SNNlYPsS5fPTFhIa/gbBPmDBE07kULts66Tvc0Yo/2yIfwV1DV34bcWWVGi01XybIHBl9qbvlfBKAUrFEcv3ZdpHNhuZw6IkZY8xoYxo8iF+toLYeGX+gFxqCZUUOO+NBVmb0hVL1LI6bU66IrMOQDkFurqpvoraBTJqwZgXvMPceFUkcVIH4Nm7cK1X2WyMtnSw2XBpLMT8C4V mchangir@indraprastha", "key_options": null, "keyfile": "/home/mchangir/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mchangir", "validate_certs": true} 2019-12-21T15:39:19.595 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286', u'name': u'sidharthanup'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC1OVCo0wMITAp8PNUqsTRPFz0SuBL0r1RzKFJUoxQm7gsP9qttZUQRlfiuZ7YRzkuGLOEBngPuIcV995o0lZ0+LVHA8jxWEJamb6sM9YgnonbGKDHfl4lT8LJd4YtjcWnMGvUUdPiPgIkfrpxXtx+RzJ7eCTmgRjAJziZrRh3AqXuKty+s9MDnmK8EQ2smpLdkMtZG5W3U2PiVaGgq2FIIOAoqr58xGqFgZ4tIpvWFpNxVsDZXIg2NdC+ouuKQvuJRWq9eERRHwq26CR+kDFNZeYSyhYKxNBqauLCsAoXNFCJzrdX5Oi9Ox4lJTsEtYX+SMziVipQS54F7MnRAUccukQ86IbXz8C5Ok2EzNo6smwRZ6UPKRvebWjIvfbeR/iwPRnrWcNgjGcaTR6yhlRJh8pBkW84nyY/bJa4WXsoNLgutdsYkjl18KKhbBldjdf0ovfL7FVgqYfAo/wpQYJJeMebP5QAczxvZIegjHRrAvbs/nbQTmuAkes3WL1hhBjSTwyIbWCzAuBJk8MNTmeZkVBvub7XoIyIySppfoJC/1a+ZsxGkzAlgrIxXPYrrBzTtXd/9YRiIgCYx3UtfdbPkE6hT6GblH6ST6QESfWyyvfGxwlOmu+F+icRyVVJbgM9T/PD2W6I04u2/1mcfoHwaQGdwvbbBkbjWzm+zwNSisQ== sanupkri@redhat.com", "key_options": null, "keyfile": "/home/sidharthanup/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sidharthanup", "validate_certs": true} 2019-12-21T15:39:19.887 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31', u'name': u'varsha'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCkeYXWjU7wVdODC32U40QRFgFH7ZEuEsOeWWaKx3Y5EBuAcwkTHiF9Bq/2mrDu+c9W46S8L38Ogc9GZ0DcCXtN/Z4z8iKOIIi5U1GzvwdbKuiAPCd8bdYP0vo0MOIRgylJo5hcnB5BCwCXRN1hmci1ES5UbSeQg0rydMiyYXHZ64AT47+qVUpn/mVP6PVw00pPqt16l9YxhbPFG74Y/WnY4FyrqXGQJfv0tcP7+UR7GQdFiI4QQV3ZUSRHRLWGcmoiiOmGimLhtzEmP4Jzy4gF4wg9q8jyX54NjfxDRZs+YuaoqqWKz/HSz5ic7W2RGKHrRby2N9wNu19nmC1NzIn3 varsha@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCW58OUR3pniaYgvr9xAExeNm4eMxgaJfFbEfEYPztTWv/xxqqihBYCeFltfgWbu7ERTy+hVNo1PJWyyczjl5Ivc3FdRmhykosygs7X74PPh02c7Em4MsSTr7SSPNN20rkJbWqiYMMOBbqbiZ7JXiSUNVilLHgZqZBEQc5Rf5I1PyBLHHJ2ZruPKnDPHCRLdwJ+MdXy2haKmGrQLhtzIKaDzh8VkfojL454k0aHCIh34Z0oBdP9757mNuYJUVEkKZF5t2oML84lWGMgwZxUWN2yO9ALTti4tNH1fXA6NipS9ztE+3hEJe1fWXEoUQDX9JNu4qEHTeCz0secQMkyZVDZSQk8yDUQxQ76yiett37sQB1WQR/3fclcahmqQ4dUS7M3Onyz6k6zia896PBK6G48TBUbFa1rw5Hx7WfNZluDMkv4YxkQvfz7XVzZx2/lhs45Kjh1UR9n7LaW/NID98vokcvASw1N6Bk/6pOd/45iPyEJemwovnQLs6ecwlMIikM= varsha@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDcQ8xMjwVf+KiFIQARcbFT+wg4NHysBRrFxk3Bx5X2k2hGKcEYR2tY/ApHZYfJ6me3LUvJEXg2c/6xa6ibVTUp/90l4y0cF6RmF5uB45J0xuBYoFJd8xJkVrhDIlZix4aFfb7TrmJEqjleBrkTwam7Q9aE3lss16nVOxU4hiViYiCANJ45wobK6tz7y5btfzrHE/iYpwniDvzLwK7EpkEK336VVd6f9wV0vG+ErEFxnNVxheXLLm05LuvdVQvFqJcTFrc7/GZ54yKg5IgwVwXqSAyOXIhgduUqk+oN1ciA7jnSqdiOKQPS/VmYkbhw2YP85l8wGVreIYUmOG19QEyVLB82oB9KS1LezM2KCH61XW22+85dHf4PvuL9DULhGt6wBCvpO10OHxw0JvHXbEgBPhiyd9j2yJmcQozbRPth/sMUvLKNXOGabZhhkPBYL7xZICHc+UwxrXs0h/j9jOjLyHzBAWMomF2lzgbvtzjcbxJ1a5oclJ6IW8yHJllkOh0= varsha@localhost.lo 2019-12-21T15:39:19.888 INFO:teuthology.task.ansible.out:caldomain", "key_options": null, "keyfile": "/home/varsha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "varsha", "validate_certs": true} 2019-12-21T15:39:20.165 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n', u'name': u'sjust'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6RDgqwA0GrvE6KqRaGCgYnFhnxdi9ehjIWvfaMa1oq0CqYxUAee2bNbp/X2Bd18Vqk8iH9fpE0+qrLgAH/8BpD+MtgGikCFMORvazeUtOCF/OCDO0AMg1jvl8aTdvrQU8ZTDPXa65Avpkc9vwugcoplnwrW7lKx++XqCWsSlfHgEdyMV3wrt/ZDFwI2AS2bZBpWM6ViG1KSjXjgTi/T3hSZPY+buYvWjzoEjQO0946iI3iiKKCGohEpCUnG02oJXcS0vvgl8dWE71451lbainEKJ4hbDhfDmy57s6VXfVUANsczwiPWiSC/ZmMlDaCCBeKHO0Gz0xXaDXd9j4SA7P sam@Pondermatic\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCYylAUS12hF6XndDQ3CkrUddKmXNSd8pOlHwO+pWaDTJPKxwtPL6sM3aOtaAPLrGkv1zD2k3iZaxmDwfPYGaXOyb0KS5ZCddtOyNXj5JLrjxh2niqFPP8SIkdHLOP+0Hv1/tB0Jz75RV+tNIT1SJTm7OZ8wZNy0qYycITSooJ14iEgyZ7rOa3EEzbtIVpHS4+YqIuIJdV2oV5egCrJyRv4sPUrS8ERXD4QZSOlHE+W4Tf+WVrEfsHOdDLRlAXJUzhuqgWtVouiYcW5ex294z28eA+12lKaPQpE0cft/NJ6FLs41yqh4ZN588bl8Ada+yTXEaHWioQEEJMlca2jIgrP sjust@rex004.front.sepia.ceph.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC08RrDQ2M7Zjcpx7RL9e+tjv48kBgDNyTenX1QdxW22JA2JR3D3ThA/ej9HtiPWcIVz16oZ4tzrCEVwaCMfb 2019-12-21T15:39:20.166 INFO:teuthology.task.ansible.out:smXq2z08ZXYqwiBuoXJVshU7lIKVqAtadT4F6Q9eQvxEo4zWdBib1W0/cqNLlYzU/DuwlUqYnWBDLGxc4fD5Kxrz366oWUDB/ULCpl77TMqfTKlStCgGXot0zLqiscqzq1W7XsuA7A5vQw/JTq7MvIME+SkUuGWX0WKwTMIRZKNWBGmlf51bNru3GmmAnKBUFsJv9H4G0FPqGY/aH02rr5dAgTTmfKt+XbWRfMTMpjJqVRGX03+1mQNoYQ3kNDz0wl sjust@office\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA5UMPySf/h9r2/OryRXrO9Qq9k5J9Dn9nvJYyaYCE82H/qkBJGboAJgyIXfbsGYO5uPM25d3fGRH4magzAQm2hGWNbbt1YWn9HaIyKCCXo9JX600/dv3OZbHyGGmgXPa4Anl16j1c2vxRRt2+ArjlyyhP7cAcRkilUGdagK9u6ALrwnz9hhXYV+BmXrxMm7zZCmdWHcWz+YxoA65es4+w+aLvVZWmUCZLF8uZrcpPSw+oK3SfZE8JPRc5vWulXbWEEfIoHb9AszPIpSyzbfPWXelmc3XlG0KD66t7w+AnNlQPt3JVePRCfFHPYeTj9B0k8sxPyUqTQIl5pu6GppRMOQ== sam@Deepthought", "key_options": null, "keyfile": "/home/sjust/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sjust", "validate_certs": true} 2019-12-21T15:39:20.455 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'deepika@thinkpad HXRUSerxILmV0ejAD/Zt1g 29b44e741567e7a2e7b664d6533247d79851fc302051c9ba34103c63a2fcfacc', u'name': u'ideepika'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ideepika", "ovpn": "deepika@thinkpad HXRUSerxILmV0ejAD/Zt1g 29b44e741567e7a2e7b664d6533247d79851fc302051c9ba34103c63a2fcfacc"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCz1UnJy64hXJKl1OsqRPlZ84m/fyHE8/Hqimy5s8sWcvRs2tL5oK30zFataqv1BC7r46+9t1eGUIRAuBg6QPuJnJCnPzS8yTVJ6DqSDGE2NAnpcMK5fSb+qgQ2adHFkjzZ4eKZbnDJMjYBtpUEp1Ve2ncdq5YCy6hGLIstkGq331ULBc/Lm3ZHBwy5I51S8eldkuP975otBYSPZaaN25QDncZzrW36zb3VwoGUAANEfHC1kLgTTFjx0bdsAGYCduXzyyXls8qSv9gbzcRzZZoH5B7tj8VDBoHszPnqGjGqwtWaelifYE/Svp+8fJKLlzRUM+XGNZoNcRl1/1XVhYV/ deepika@deepika", "key_options": null, "keyfile": "/home/ideepika/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ideepika", "validate_certs": true} 2019-12-21T15:39:20.738 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46', u'name': u'gsalomon'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDC0h+TpaUkm2V6Hbf7aPCTys/nsm51CmY9s+bdwEXAiSAWpq/7Xi2LlN0mZgsrAYBHbr5L+Pz8x0YdigvWysUK6U9XRSPM28cbWmh8F25re9HEk8ARESkMZKTv/hKFs1v7kTHbT9hNNynbQGQOqEaazjr0shcli+Byvop27riX3QDDSsd5xfCjjyDz9fiA9YESlF0kbXyFUKPsoaBhRv9FUhtH6DxlygxtxTsKyKgeoiKw2sCfeUMckcJKnGCMtkXW4U8h7NL/fPZFyBaMDRi/TtoBKaVGLaWqF7RuukfjoMCwhuIDmuTG3WhanH4HJrfNenPZkpZUBhzWwZcleao8SD3mXXfZ9gcZb8QB2ymEtTqrAEyDoNWO4q8ljNP3WJXIgF0h0Sjmx/CV/i0ArEYlBMwBD7DFVnPJqCqSyypuQdDGsDNQZxaRCiYg7nIslFOpDr+0PLp78m+CHyYbSFLTvVVJoJ1k0YhMnDPHM29nt7fMGx6Qrigwnuk0yKg1bfgJ/t0TOqEw0bqGKIvHnyoP/n0ObazEsiPl5pRr2ehs/Rqf35Iwhc1i02Hhz/3Sqwc1eAgqm4/OVghxqUJGpRocua5Gf/vO32wwoyGxh2sl7y6K2pNDA3glhKYbZeIh1fjnaTjikgleqIQkCBxbTeAE7n5+T6TcFtnII6RSxYv3tQ== gal.salomon@gmail.com", "key_options": null, "keyfile": "/home/gsalomon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gsalomon", "validate_certs": true} 2019-12-21T15:39:21.016 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de', u'name': u'soumyakoduri'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1q3k4RvLLCaG874GDJjja4eYqFwrMjsTU+aRzambaVuCmAJT3KQMGuwMjTuCIN7R1Fqr+/dxUClamsfsGqPMe0ASHT9X/0AyNgEkr6na1Fvk9NAwAFelwtiRIHhclzYf6KY8QHZN/eBy43prVKDXd30b15gwQyVqU0/xmny0GrYCN24k3N/R3divt2GVqptYgEwS2KEyFT5EMnJWjdIVG9mBOHrgec2VF/rc6w3BZ0s56iKdpOeCgk5ISYTcom+Xrf+fSDHirlviZv05tBP8ESmampUT/EnwF8Mdaln5XJvZPhrq2JpUisi9fVeR+zlzg6gRGubvokdA3AERrhCQ9 skoduri@rgw_vm", "key_options": null, "keyfile": "/home/soumyakoduri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "soumyakoduri", "validate_certs": true} 2019-12-21T15:39:21.295 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a', u'name': u'kyr'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDPeuSTD7kv8yQX6dW+7Vpq1mqrcr8JGubPYa1DpGTV/gE50d5L42UspUN2FXmICDMiV1h4Q9WQAFma3168f1I5I94qFkV+/56szCMoILClm7IW/EV4t8fTxoyOpEa9l4qSU3qgJpDbY04CkiyVJFuQj24Zq3gD/70jYgGpy8RfA0BihSR6F+A0WZLh/fESjqfor3/pFWhUCRnPEqrZsXyuB8lRgDTyGp4s1cHyUqmFMuCPDRMtok0rSVpaRZj2Zj8lboah19UI/L89ptQgEdqsOKzlkxiT+1GdyIY+jpPCM0a3yZz75l3Qiu0k6q7qUV8T0hm1GZiBguvST1KMcwnt kyrylo.shatskyy@suse.com", "key_options": null, "keyfile": "/home/kyr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kyr", "validate_certs": true} 2019-12-21T15:39:21.596 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849', u'name': u'sseshasa'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeERrQ0NVXhuTYzI08rlKRbHZVFXSv1YC7sGq6aCWRbTCv7Ad/t/3OJz3vfcd2TjBMvABfrYCFgTFWIaJBaeA3BXUUU9y9TwnzuxKv4C46WJYoalVGyLEpWpcGbJ5v+leA6Je/+lunenplb7TBKGE/YTgggEXyX7KnunYlv+Jdm33TmAZTcHrLk5QR/6FdRGKmdlLna0dXZc3v04+AI8W+p9ghAv1+e8ENxJDTF8p0MawBiKWcSj2mysOhQ2Cj91qZ0qTnLLeu0CPf4rBO+qjYIshiCuTLhYt2UazlzME4nCCsVqK7Nz9olA/lgvJLRaPxgQQ7iCMmIZilS6B4Y3oL sseshasa@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCo0zniUEXuQ4JJuuRtfnEK8WHqr97AmXEniGPY5U23MTxhMuP8CSkGWkUHdmXw0H1tDKbYgk97M6TspJropNGKljla2H1/3e2T1BOI1L2WfcsOh6VPYeyOj2ssMJeEDdQgd0J3g+4oj2QS+guPlT24UhO0CBhRZ6waVlJMgagDYyDLCLGI4ASoBYx0AOaoyjnEQtRd/gyejzv7jntkHc9YNBPsJ37X6hTYNA/s2ORJE7QMXd6xI60mTBuCdD3JcPYoUSvyCUa7rIrDc70lAcW5XtxewvMBogTZJLoFG/d6Ugij49h4AfwrtoUCtLyJOWJX6PTfomXely8Ym6KpN/3981+m70dhcHPDJ66QCWMd1LAgftVXTIsebA+Bxfg4Vsbrv7j1lJXzmMfsOl+1wVPe+Y9h28GyeIlalW3laURZGV7dhexGZ8YNxPuzxtAlWtfw+kddbjw0j4ODtgVXrj4SDxqmTTYYuMBkrQ3H33sOgPLx8DyA6e4pckb6lOOEvb41Yi5tyP7BIJlYN2PpNVDkSQd/t5Q0di4Ct+3vqLRCkPuJiJAUyaBGT4HM7yQc25+Y0B3sXNON7BjeaBXB7OwKZMp9w1jwbJt4JL/aGOw3HVljVyVgu+pIuQ2ISd9VBcmAJK5EGdWe5Q57h5fJkD+TVrXgeNTaufdclh5vWOH1qQ== sseshasa@teuthology", "key_options": null, "keyfile": "/home/sseshasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sseshasa", "validate_certs": true} 2019-12-21T15:39:21.885 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189', u'name': u'rfriedma'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDIC6uqbIg9XaTlDZd7KCX7FXSzfGFMEWXhx42ktz3ehOPLviA/yNpVd+LmYHSgzd5Szd3mMxleisiE+JdSq/q5YyPAF6eD6+wfynkpMv5aWNHGI0yVfKbCwdOX8bY6yqhyVwMDo14uoFvdDcM4Lj+vz8F75VJHUU5ihJNOW24EllobTLzR4XI6AcM+mjbXRnBLLVoBSuDaU2D0kWqMn8rDCJKdUJnoiy9oxIok8eMLfGkGsL55oOx9BflMSz/0jM9X5GLQahYiECbXjvCIarCkUFE8txNoXM9Djmg8QqhAQDhY0hzJTauFJJaD8Ax1uFQlXONed7tv7a7Tb3dH+/jb rfriedma@dhcp-4-239.tlv.redhat.com", "key_options": null, "keyfile": "/home/rfriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rfriedma", "validate_certs": true} 2019-12-21T15:39:22.176 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b', u'name': u'lmb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIXHA8BwV4OnqlaHHyYofsp5X+wanose+aneJVx2Or05PcWA9c7xFEOerkvEn9LvivSRd9G9o/S8SvnGH91tG2bbOAidVZ69FGul9OKa/BCJYUP0vVBX09/Td3n7+R2lVlvu5XiyC5SUzCjaQEZLOpjq23rqaqdCDXE6wFnxfpkmcsccbG/xUiOEBeBGI92lntdx99Sn6CxWWA27lMMaYkCRvX6ziRZEm6O44N2nMm+qYm0jmMyGv1iyQkJxtt0YaLPp+X9t2trUEzEjgBCXqqzEzBjB7T4rbzhAuvU3NRHiAoWXSnrY5AQOTTvC/wpaZV/gF2NDNThjMcpcEdNstleOctXtN4XTTJQewbG+t7I9knR00TPpsoUvUtpbcoFOluGgWTDTFa6GCmDPppx23nZgulOniZUVTI2CbScSpc8sfp1JE7Q6vLECY9lJ+GpVCbDAbIEE9WHXSkJ/6L9yf+ZoHbpqCkEFDcWLfz3B5NFoJ5gne8B8uzd6ptKVtJrP8pMvoOyydyvQlmkgh8tNDsOnp6kaUNG0NHmtfGlCTkbM7oKsKbf87Cv1HgYuZl/HP7go5sOX02LfBfbvWrawZzOaZ3dewEq74w5hVMKEHrmJPstIpcHSxZpq33f2j4Z/uKW9tJat6rgSWK1I+T3tCd++76apxbOYl5TDOaoxCyjQ== lars@marowsky-bree.de", "key_options": null, "keyfile": "/home/lmb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "lmb", "validate_certs": true} 2019-12-21T15:39:22.456 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc', u'name': u'ksirivad'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDMDUcToIhs8tiIozgPudiYyD2upEX+VvsiJRti7zrA/1HPIDARYt2+xyWCASm5x5vdohRM7wnBLUlbBVnvr6ZUQvfShNDsS1Uu7hEfYF38MtW+dVwvGlm14J4m1gbjzqVlZvO6bg6rfGo7Qer0bKUlxtqbisQ/kCg/lVWR04CNxdRtZxEpaQ21y4C5jAlrpFTzxQSuiCLEyY3EAKdZUaZElFEcXlna0ysCP8inlzhF402PkSdl9DkkPpg3IR2dUEGqsblWSdLVk19EFavjtGAAnsvaBWWm2rCLNC5RmhGQaS/Z52Aes1nBuT4cI6qLjbEMqAzLwxiOR7l4IPMpK5EcuRK+FaqQG0wMAjxPc95CmxDzi58ZRJ5ERV8vgps73F1AJXWU9Lb1P77BTsLXqiwko2mvPf6e0LwCxQuxYJWc+xvO7Ji011B+nm5eLCCC/LcDtTPjkimTh7nkvUNi/yTKCXi9KwLISAHHd335KnfrL28rU3GahMXs4NjOxuN8eKM= ksirivad@ksirivad", "key_options": null, "keyfile": "/home/ksirivad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ksirivad", "validate_certs": true} 2019-12-21T15:39:22.745 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4', u'name': u'ofriedma'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC0POsA+0NyxecLo3Y821G6W3+jJPKFMuLSIl4FSLi25vuyuM5+mFGmCNWv8gg93eQhG9Y3SOygqmkaxzxDtBS/V6VNC2f8hZsVw9B+HbnfggvCNQSgjJ5rhvf6IHfx/aHq4IHF+stqdcoWhwR/68wIhGNqtP7Nhqnzk2aN1kG9bU2iVk4CH8XGIwqwET3aWNuCMcHi2AhrvDBpZu7bzo/6ARUJDqEfTEC/XIkzv8/mEkJ5igcycARVy6dChuCwfZc8rhfwQpekwx1Y9/WBFdNQlYkN4fVe8yIqT5VsQ3AcRv21GJk8L7mwB4hF9dVAi4p1cSyxatR3kDifsFc8VQk9 toor@localhost.localdomain", "key_options": null, "keyfile": "/home/ofriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ofriedma", "validate_certs": true} 2019-12-21T15:39:23.033 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52', u'name': u'h4iresis'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDc/8sWrMq+bqKBPmDPplUgdMoY2SQL+57jRSh27PAoYW9rNVR/UgK7EA9oxFLfoi9wJ9+VKU4opeoohOXLWQXXatLvfXjGE3fRYIonQfwJqKrasqQJTGWzRZ9t1/2Cb0VHkoAvipCJzWmgB46X6q73XW5Td1H3b+AINd6E4Rsm1QjBOhk0DMjCdfwrVUZIfGdsEo+5u/AwioHwH+NDJvXTWT8ZigeZBNHJSY/EQn1+15ut0QB2rvXkJasvvDlVbEsPQtM4C3VX1ronmAnKMvCp1O4EtAXaoKSOx7XEN++UAvVn7czDCWk/UFgpzZOqTcQJwg82XtS4sZP23CpNe0G9 h4iresis@fedora", "key_options": null, "keyfile": "/home/h4iresis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "h4iresis", "validate_certs": true} 2019-12-21T15:39:23.322 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5', u'name': u'shyukri'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCqhWVTosH12864VGY5S3/k0yKc59GbV0m2WxD28gAf+uvJ1/y8gekD2wDzBv0DWqEekYLV7L68gpapDXyGosQeZ6n5OO/NZFZmW+Xq1P7P1QHJil6kVpwk927lzX440pwNf3+PRvTbLzlbuVFWJRpiykv+juJyIFjzXLJRF67wB2qgZVm5cRs/GOi4fnvTaou/Ae51mvsO8M3JL5cphApFKD/AO/X+3pHBhLjvaW33cYsp0Z4Rce1nm8wxSmvuJRF+HAfjBlknhDl5ZaJGA96RYA+g9WbAXimFprg4xeiRp7liPExQwtVgackJweoC4MahC1glmjC3eOh9dvrrxLnALOLMOkyCSK/tOF/iq2PJaLViObanPI8mnNkMLbdJyzUBf8hRGdSLcw98qxaf0sSZ+vz4VQS6e9CwX7ikkNNsTeYXjnJ+1yU3mfCW1CXJDr7kBDVG2OTLefCpMY8WBAYDiEaK0RW83cBcFIKfecKYd0qStHRaOsvmyWdFYgBh2vM1UWjro5WXlb+7aJPrUp8ZruapkUXUwhjHA7FwGA4ysE0tsOVVFm0Dw8tvyVVkVNm/9EObG28dWctioZ2+ZEzTWgMJU26Q+wOJ/U4qZyIvpMRUgPZc5nM9J8kcmvNWtb6DV9k/6p8h28kiOxPkxLtPP+3TlBCHjq//mhEnJZ/d8w== shshyukriev@suse.com", "key_options": null, "keyfile": "/home/shyukri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shyukri", "validate_certs": true} 2019-12-21T15:39:23.604 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e', u'name': u'scarvalhojr'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxDJtmmcnrbA4AUsmBBZq/M4xNdIbOT+cQQxhUKeCu8F9mht+RznwyaaJbVIlD+UNv9LMhRyzI5oktSalokNltlrtPQMoP52PXH7KFFGnVnexCHAk4nGvEhuZCniO5T+ENm2wFR47lFEQK7lyJJLp5PFiSM7vM1MK6UkMvRmCTRildjaYLyf3EEvBXifbsGmR0tBIXtml6KITCAaT6l0E94055oEGN2G5NhZIvfLcojt5liEkXEHQNdfkUDXM0Go43JZzgsSWDXC/7RYLrAJwz5mq4qCa9YZL/3Spjso37+1oLq7lOQ0TDJUAQXmnW0So/NkzECx4MolVcylhTzmgj sergio@milestones", "key_options": null, "keyfile": "/home/scarvalhojr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "scarvalhojr", "validate_certs": true} 2019-12-21T15:39:23.892 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b', u'name': u'rakesh'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEOigBGMeElb3uHXX36HxLtPJMG+iG/6D7NDFCVR6AyYsmimBQb6SlSIR6SaFULb6HT/IFVk0Jf/z0HoipAP7Xu+wvP/J6LFoMdguxylh1ov8raqxQJrGLl9n64EI+qWINwEM0MKMUnDQ3tmMKBl78LKQVQTYBx1+9cXsT6DE1yhgmV79XIvOOb+v53agSi/kLJF2i9K04kpi1OejXg95Ub2eLMY69Jm8PnLy1Jq8DhC8pz7nrfzbKtOC7HbytTAO1XTlb90lVhqC/1yTMAUc/rpJwoSEHc2epGeZWdMsJhqhX2EqMwcV53M3qcFSGreGNJ/Ncd4/JHxz7q0IU411t rakesh@localhost.localdomain", "key_options": null, "keyfile": "/home/rakesh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rakesh", "validate_certs": true} 2019-12-21T15:39:24.175 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f', u'name': u'ryg-'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvRwrsN62CReT0/kOeFEXGA3THhzJ+hwZVpUzo7VMRZeSo6LgNlVGdrQchMHnlwBvuwaVNJlBAkQr+5J5BB3ikrmbcnLHfPutXBBBB6uDfSwwbqjzUY7elTbz9ul6AkegR9K65O44ouD9+WC76D+Tp0maqnP1w0O3h1GVxDHe34LpEwBvZJ+/BSYmBpwRZqeXFRADORPzdtW08PSiRErGHb8j+j5i+br37JfOOh+A/9dHwQw9alBVL2DijwvrPMa7dV5yX00dlMSTL2LLo9IdI4aFPmie8oic0eanFbQDq210ilOR65IGeshKHEKOq0KFFz0nSK9OdO03OkSEN/0d5 ryg@tapir", "key_options": null, "keyfile": "/home/ryg-/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ryg-", "validate_certs": true} 2019-12-21T15:39:24.465 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3', u'name': u'prsrivas'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtVTFNHORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@redhat.com", "key_options": null, "keyfile": "/home/prsrivas/.ssh/authorized_key 2019-12-21T15:39:24.466 INFO:teuthology.task.ansible.out:s", "manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} 2019-12-21T15:39:24.748 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'zdover@slim_thinkpad H/78rBqDIJ5AAjMT6sTmKg 49fb166c553d14549491280458269f987bea97295dbf525ffb7c35336175ee42', u'name': u'zdover'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zdover", "ovpn": "zdover@slim_thinkpad H/78rBqDIJ5AAjMT6sTmKg 49fb166c553d14549491280458269f987bea97295dbf525ffb7c35336175ee42"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDlnlBzRdWRqns/9VrYuF9tLILUjeGd9z8Tf/dPZmxAt3nnZqpTvI2xXemQpyt/2lsD0U24EuTc1Y/dBsgqvXtZOw51TZE8RK26hvCCaKGfcKONjvR7goHAmkYPHK9+F6m7fgYkRzA0+ZW4HcWNh/xxhhL2G1EGfz5bzIoGQJHSdzc6VeVIsY7HiYEMTxvxdsgV72Kjq97BdmHm56RBzoH24oN0UzNZS8Syx5SuWj9dA3IyFBPwjLlozSk6U2Ofd7qaXuQKl807eD9g6RhRldzHPS+i0EK3EM6Jvho5Y08k1KqV4gIEwMdzDbD3SH0xq+nUPFKcDTUFNp+tkg10GxqN zac.dover@gmail.com", "key_options": null, "keyfile": "/home/zdover/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zdover", "validate_certs": true} 2019-12-21T15:39:25.029 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'rraja@thinkpadp50 8S3Lez75woSBdYiLqQxf2g c9d1666dfc7a3b762700cd250f775a506c397b0f0e6a86289ba555a3546e532b', u'name': u'rraja'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rraja", "ovpn": "rraja@thinkpadp50 8S3Lez75woSBdYiLqQxf2g c9d1666dfc7a3b762700cd250f775a506c397b0f0e6a86289ba555a3546e532b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC4zwWf+kMEDgaSOrX7AlD+UCOgf4/fRiorQVJqC8GfMNynfiFVZaThngwIrwtQROfXHbzspi+AcRvDLewY25mXyuxVgOfSbuSIEqiUNvzcVq9IItI1J58xeDBpIktAsoHHS6JENV56fAT6krzRAQLnSa1k2b0pyKuPBzFSeNgiaQMhjrCLAaCi9yLsdKpnWgXhyfrdqvN2r/Ctts/zlTPqtRYdvdIL/p/l+vg7Q8S3OhuAGtimaEq3RE70xadaPbGhsLISMTrbfYXbxw84KKZpNYwO4zmQ2ztcvC9D7A/8ov3+A5Jq42G+pq7ORuix2NoTWtADMiwvNe1HyzS/7cX/BuXLFssRY1bzRvp+x2ExZ7b+bFx+TE/RduRh0eyDNLg7SyL83aeTr2Usa+mPoCAMKed9XtYbRnUu7ox4wmfeAgx6jM9uG5uQfauJ1XpDddrVBG8KYBvSd0BM2cyRdcAXqwxg9AVPZcdE7sbBk44CGVhw+ziZ3e3cLBTO4y44KRQre+1Bhe/Ze8OTQWLFugl/XagFlJE/L6fGUIPuFojmQvlw5dBZwtUURnxsu8yHm3iIsj4y8nR1zHMj+zuMk6rF3ZW0epPAECAdq4gG8X2wv3HTq0vLBkKpi1KMAmntEtcVwF0fzMt5XYqRd4jceFgonAAipJNDknHgi6yaUzBYmQ== rraja@redhat.com", "key_options": null, "keyfile": "/home/rraja/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rraja", "validate_certs": true} 2019-12-21T15:39:25.360 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f', u'name': u'toabctl'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2mEyxfkW1u84u5BocLCAXESj2qObWkxUd20cmPolhJYlIWLOafa69Dw5A9CjFBp0U6Jx5d0JWcwmAc5ZeffQ7t7vzmtHCZnpKkyZhN9bu6vuCh4N+I643SI/K7tdbB2E8ZE6uI7ORMAan5+t0bNSVolUVq6X1w7dIA5El54K10WNdiwgRKW6e3anNbJ3rBLGEk7gawZqDDOqTvVQmOBGWek3BjSReDvg3rkmTQLxJkiKdxnx4/s7gzH1xTpsS/OludIuSxF0BK064ZcT1t09qcjnT4Os1inV2GfYswt+UhpgqrHqXjEs0Z3y4Oy77QBKL1rDf93CP4a4+eJjqXe+v tbechtold@suse.com", "key_options": null, "keyfile": "/home/toabctl/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "toabctl", "validate_certs": true} 2019-12-21T15:39:25.653 INFO:teuthology.task.ansible.out:changed: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'mgfritch@laptop 3KhqWETFJaVsH/I8FOBfUg 0aca6c3c6d45c554bad8ba9853f0bc575c9915eee33ee6be2059568ad09a9d72', u'name': u'mgfritch'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 3KhqWETFJaVsH/I8FOBfUg 0aca6c3c6d45c554bad8ba9853f0bc575c9915eee33ee6be2059568ad09a9d72"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCitWIN7jm5KEmB6jqcA+ufKRLodxkJ1GIdaaueCdeXdMHgWNH9FUDCUJk5dcA11MxQO15DRH/cdhb3QHPGiMv4XrBeK6HHYwGttzkcUEz1EDUhpowOrnFETbPa7K39ObmRcgdvoGXOSrNZ4mvptCTz9Xb0MF+Gcg5iJ4g+qZzIVa2/ueecYY8ZorbaH4m/NWUub6Rf01Q04HfxTvKrUFswrbCoE/Aigz/Oh88RiS854XoTvV7rE1tKbyKkaungwMAaQSNnZkq4hi8nbp6pzTxJLV7N/ZYMhVpdxSbWcghFg6cy7aeoCzZMN97WcDhpdfbhRT0dbqCkMmHV6t0usDDnOss2v68UCzkxuA1Hj4uy8rRHjXBtYqI6xsrwCjvmgP05B14TpmFdG+3GdRHx2+QlhnGj7nuPf1KoWW/f3tD85DXoEmVl6RJ4KTrMeEVlOn3kdMLVH0y2hIqPTYMP5cMVNi0EdfeoBdd9kxMKMvoSVBAgmLzGVv17o1lNvVS6V1kyt7gME2PfNR2Np+U1cJVJfUJRTm28mHXBVoDXG9s0zv8+LpQ/gVPS+vsPAfxWWYGW8ZmEV3ZorDOZycSPybjHkPFzDAdUhPG0Db0o+F9ug+22F7pvifsdDZ42zqGGLbyzfIQE3AtyI7LR7Sc5tpN9rKTeSBYJ0yNcDKEj5FUHDw== mfritch@suse.com", "key_options": null, "keyfile": "/home/mgfritch/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mgfritch", "validate_certs": true} 2019-12-21T15:39:25.763 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:39:25 +0000 (0:00:29.824) 0:01:03.771 ***** 2019-12-21T15:39:25.810 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2019-12-21T15:39:25.811 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9', u'name': u'andrewschoen'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "skip_reason": "Conditional result was False"} 2019-12-21T15:39:25.812 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n', u'name': u'zack'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "skip_reason": "Conditional result was False"} 2019-12-21T15:39:25.823 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd', u'name': u'kdreyer'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "skip_reason": "Conditional result was False"} 2019-12-21T15:39:25.833 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'alfredo@papaya.local V5VkRT2Du88WwQrYeK196w 4dff4d9bdc4d9ab60532814c338abe4534cb45a9ca729bdbc2411ad513516b47', u'name': u'adeza'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "adeza", "ovpn": "alfredo@papaya.local V5VkRT2Du88WwQrYeK196w 4dff4d9bdc4d9ab60532814c338abe4534cb45a9ca729bdbc2411ad513516b47"}, "skip_reason": "Conditional result was False"} 2019-12-21T15:39:25.844 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\n', u'name': u'dmick'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\n"}, "skip_reason": "Conditional result was False"} 2019-12-21T15:39:25.845 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n', u'name': u'sage'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "skip_reason": "Conditional result was False"} 2019-12-21T15:39:25.856 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n', u'name': u'dgalloway'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "skip_reason": "Conditional result was False"} 2019-12-21T15:39:25.867 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045', u'name': u'akraitma'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "skip_reason": "Conditional result was False"} 2019-12-21T15:39:25.881 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e', u'name': u'trociny'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "skip_reason": "Conditional result was False"} 2019-12-21T15:39:26.152 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007', u'name': u'smithfarm', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "key_options": null, "keyfile": "/home/smithfarm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smithfarm", "validate_certs": true} 2019-12-21T15:39:26.163 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n', u'name': u'mbenjamin'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "skip_reason": "Conditional result was False"} 2019-12-21T15:39:26.444 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01', u'name': u'aemerson', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt", "name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt", "key_options": null, "keyfile": "/home/aemerson/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "aemerson", "validate_certs": true} 2019-12-21T15:39:26.726 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78', u'name': u'amaredia', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "key_options": null, "keyfile": "/home/amaredia/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amaredia", "validate_certs": true} 2019-12-21T15:39:27.021 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5', u'name': u'tserlin', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "key_options": null, "keyfile": "/home/tserlin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tserlin", "validate_certs": true} 2019-12-21T15:39:27.043 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28', u'name': u'dis'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "skip_reason": "Conditional result was False"} 2019-12-21T15:39:27.053 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n', u'name': u'gregf'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "skip_reason": "Conditional result was False"} 2019-12-21T15:39:27.064 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n', u'name': u'joshd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "skip_reason": "Conditional result was False"} 2019-12-21T15:39:27.065 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n', u'name': u'davidz'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "skip_reason": "Conditional result was False"} 2019-12-21T15:39:27.076 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'gmeno@gmeno-virtual-machine FKFu8B2pMqotpmEVAO1few 8229574e499eaf767a408909f5afdf2e2a0bb8f3e61b18d63a651f7102c68dbc', u'name': u'gmeno'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gmeno", "ovpn": "gmeno@gmeno-virtual-machine FKFu8B2pMqotpmEVAO1few 8229574e499eaf767a408909f5afdf2e2a0bb8f3e61b18d63a651f7102c68dbc"}, "skip_reason": "Conditional result was False"} 2019-12-21T15:39:27.086 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1', u'name': u'ivancich'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "skip_reason": "Conditional result was False"} 2019-12-21T15:39:27.097 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n', u'name': u'wusui'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "skip_reason": "Conditional result was False"} 2019-12-21T15:39:27.109 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304', u'name': u'zyan'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zyan", "ovpn": "zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304"}, "skip_reason": "Conditional result was False"} 2019-12-21T15:39:27.120 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH 5ivdxgFO4eIkbXVhl8xkvw 59212d29b8b42d9fe457c1b2c43d774e1d25807be10dcc1252d4aec63b97a467\n', u'name': u'yuriw'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH 5ivdxgFO4eIkbXVhl8xkvw 59212d29b8b42d9fe457c1b2c43d774e1d25807be10dcc1252d4aec63b97a467\n"}, "skip_reason": "Conditional result was False"} 2019-12-21T15:39:27.131 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n', u'name': u'tamil'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "skip_reason": "Conditional result was False"} 2019-12-21T15:39:27.142 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n', u'name': u'jowilkin'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "skip_reason": "Conditional result was False"} 2019-12-21T15:39:27.153 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4', u'name': u'bhubbard'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "skip_reason": "Conditional result was False"} 2019-12-21T15:39:27.153 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n', u'name': u'yehudasa'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "skip_reason": "Conditional result was False"} 2019-12-21T15:39:27.164 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4', u'name': u'dang'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "skip_reason": "Conditional result was False"} 2019-12-21T15:39:27.175 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb', u'name': u'branto'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "skip_reason": "Conditional result was False"} 2019-12-21T15:39:27.186 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6', u'name': u'xiaoxichen'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "skip_reason": "Conditional result was False"} 2019-12-21T15:39:27.188 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e', u'name': u'ffilz'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "skip_reason": "Conditional result was False"} 2019-12-21T15:39:27.199 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n', u'name': u'joao'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "skip_reason": "Conditional result was False"} 2019-12-21T15:39:27.209 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n', u'name': u'nhm'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "skip_reason": "Conditional result was False"} 2019-12-21T15:39:27.210 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313', u'name': u'nwatkins'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "skip_reason": "Conditional result was False"} 2019-12-21T15:39:27.221 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22', u'name': u'mkidd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "skip_reason": "Conditional result was False"} 2019-12-21T15:39:27.232 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af', u'name': u'jlopez'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "skip_reason": "Conditional result was False"} 2019-12-21T15:39:27.246 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb', u'name': u'haomaiwang'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "skip_reason": "Conditional result was False"} skipping: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608', u'name': u'jdillaman'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "skip_reason": "Conditional result was False"} 2019-12-21T15:39:27.256 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3', u'name': u'kchai'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "skip_reason": "Conditional result was False"} 2019-12-21T15:39:27.276 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2', u'name': u'vumrao'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "skip_reason": "Conditional result was False"} skipping: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n', u'name': u'dfuller'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "skip_reason": "Conditional result was False"} 2019-12-21T15:39:27.284 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n', u'name': u'owasserm'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "owasserm", "ovpn": "owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n"}, "skip_reason": "Conditional result was False"} 2019-12-21T15:39:27.295 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n', u'name': u'vasu'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "skip_reason": "Conditional result was False"} 2019-12-21T15:39:27.297 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37', u'name': u'smohan'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "skip_reason": "Conditional result was False"} 2019-12-21T15:39:27.308 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114', u'name': u'abhi'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "skip_reason": "Conditional result was False"} 2019-12-21T15:39:27.318 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611', u'name': u'cbodley'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "skip_reason": "Conditional result was False"} 2019-12-21T15:39:27.329 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195', u'name': u'mwatts'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "skip_reason": "Conditional result was False"} 2019-12-21T15:39:27.341 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6', u'name': u'prsrivas'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "skip_reason": "Conditional result was False"} skipping: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea', u'name': u'pdonnell'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "skip_reason": "Conditional result was False"} 2019-12-21T15:39:27.352 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868', u'name': u'jlayton'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "skip_reason": "Conditional result was False"} 2019-12-21T15:39:27.363 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f', u'name': u'rzarzynski'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "skip_reason": "Conditional result was False"} 2019-12-21T15:39:27.364 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e', u'name': u'rdias'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "skip_reason": "Conditional result was False"} 2019-12-21T15:39:27.374 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953', u'name': u'vshankar'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953"}, "skip_reason": "Conditional result was False"} 2019-12-21T15:39:27.385 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58', u'name': u'akupczyk'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "skip_reason": "Conditional result was False"} 2019-12-21T15:39:27.396 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e', u'name': u'nojha'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e"}, "skip_reason": "Conditional result was False"} 2019-12-21T15:39:27.397 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43', u'name': u'ifed01'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "skip_reason": "Conditional result was False"} 2019-12-21T15:39:27.407 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4', u'name': u'myoungwon'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4"}, "skip_reason": "Conditional result was False"} 2019-12-21T15:39:27.418 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2', u'name': u'gabrioux'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "skip_reason": "Conditional result was False"} 2019-12-21T15:39:27.430 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f', u'name': u'leseb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "skip_reason": "Conditional result was False"} 2019-12-21T15:39:27.430 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca', u'name': u'hchen'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "skip_reason": "Conditional result was False"} 2019-12-21T15:39:27.441 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n', u'name': u'jcollin'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "skip_reason": "Conditional result was False"} 2019-12-21T15:39:27.452 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc', u'name': u'xxg'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "skip_reason": "Conditional result was False"} 2019-12-21T15:39:27.463 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53', u'name': u'pcuzner'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "skip_reason": "Conditional result was False"} 2019-12-21T15:39:27.474 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2', u'name': u'mkogan'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "skip_reason": "Conditional result was False"} 2019-12-21T15:39:27.474 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed', u'name': u'vumrao'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "skip_reason": "Conditional result was False"} 2019-12-21T15:39:27.486 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736', u'name': u'henrix'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "skip_reason": "Conditional result was False"} 2019-12-21T15:39:27.497 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26', u'name': u'pbs1108'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "skip_reason": "Conditional result was False"} 2019-12-21T15:39:27.498 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315', u'name': u'clacroix'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "skip_reason": "Conditional result was False"} 2019-12-21T15:39:27.508 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006', u'name': u'epuertat'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "skip_reason": "Conditional result was False"} 2019-12-21T15:39:27.519 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461', u'name': u'tdehler'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "skip_reason": "Conditional result was False"} 2019-12-21T15:39:27.520 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305', u'name': u'laura'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "skip_reason": "Conditional result was False"} 2019-12-21T15:39:27.531 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce', u'name': u'yaarit'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "skip_reason": "Conditional result was False"} 2019-12-21T15:39:27.542 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3', u'name': u'rpavani1998'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "skip_reason": "Conditional result was False"} 2019-12-21T15:39:27.552 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4', u'name': u'rishabh'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "skip_reason": "Conditional result was False"} 2019-12-21T15:39:27.553 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2', u'name': u'skrah'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "skip_reason": "Conditional result was False"} 2019-12-21T15:39:27.564 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8', u'name': u'smanjara'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8"}, "skip_reason": "Conditional result was False"} 2019-12-21T15:39:27.575 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41', u'name': u'bengland'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "skip_reason": "Conditional result was False"} 2019-12-21T15:39:27.575 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d', u'name': u'pnawracay'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "skip_reason": "Conditional result was False"} 2019-12-21T15:39:27.598 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8', u'name': u'alfonsomthd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8"}, "skip_reason": "Conditional result was False"} 2019-12-21T15:39:27.599 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e', u'name': u'oliveiradan'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "skip_reason": "Conditional result was False"} 2019-12-21T15:39:27.610 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba', u'name': u'swagner'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "skip_reason": "Conditional result was False"} 2019-12-21T15:39:27.610 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e', u'name': u'yuvalif'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e"}, "skip_reason": "Conditional result was False"} 2019-12-21T15:39:27.621 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852', u'name': u'kkeithle'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "skip_reason": "Conditional result was False"} 2019-12-21T15:39:27.632 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130', u'name': u'emmericp'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "skip_reason": "Conditional result was False"} 2019-12-21T15:39:27.633 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0', u'name': u'mchangir'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "skip_reason": "Conditional result was False"} 2019-12-21T15:39:27.644 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286', u'name': u'sidharthanup'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "skip_reason": "Conditional result was False"} 2019-12-21T15:39:27.655 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31', u'name': u'varsha'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "skip_reason": "Conditional result was False"} 2019-12-21T15:39:27.666 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n', u'name': u'sjust'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "skip_reason": "Conditional result was False"} 2019-12-21T15:39:27.666 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'deepika@thinkpad HXRUSerxILmV0ejAD/Zt1g 29b44e741567e7a2e7b664d6533247d79851fc302051c9ba34103c63a2fcfacc', u'name': u'ideepika'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ideepika", "ovpn": "deepika@thinkpad HXRUSerxILmV0ejAD/Zt1g 29b44e741567e7a2e7b664d6533247d79851fc302051c9ba34103c63a2fcfacc"}, "skip_reason": "Conditional result was False"} 2019-12-21T15:39:27.677 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46', u'name': u'gsalomon'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "skip_reason": "Conditional result was False"} 2019-12-21T15:39:27.688 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de', u'name': u'soumyakoduri'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de"}, "skip_reason": "Conditional result was False"} 2019-12-21T15:39:27.698 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a', u'name': u'kyr'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "skip_reason": "Conditional result was False"} 2019-12-21T15:39:27.699 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849', u'name': u'sseshasa'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "skip_reason": "Conditional result was False"} 2019-12-21T15:39:27.710 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189', u'name': u'rfriedma'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "skip_reason": "Conditional result was False"} 2019-12-21T15:39:27.721 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b', u'name': u'lmb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "skip_reason": "Conditional result was False"} 2019-12-21T15:39:27.721 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc', u'name': u'ksirivad'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc"}, "skip_reason": "Conditional result was False"} 2019-12-21T15:39:27.732 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4', u'name': u'ofriedma'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "skip_reason": "Conditional result was False"} 2019-12-21T15:39:27.743 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52', u'name': u'h4iresis'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "skip_reason": "Conditional result was False"} 2019-12-21T15:39:27.754 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5', u'name': u'shyukri'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "skip_reason": "Conditional result was False"} 2019-12-21T15:39:27.755 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e', u'name': u'scarvalhojr'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "skip_reason": "Conditional result was False"} 2019-12-21T15:39:27.766 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b', u'name': u'rakesh'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "skip_reason": "Conditional result was False"} 2019-12-21T15:39:27.776 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f', u'name': u'ryg-'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "skip_reason": "Conditional result was False"} 2019-12-21T15:39:27.787 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3', u'name': u'prsrivas'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "skip_reason": "Conditional result was False"} 2019-12-21T15:39:27.788 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'zdover@slim_thinkpad H/78rBqDIJ5AAjMT6sTmKg 49fb166c553d14549491280458269f987bea97295dbf525ffb7c35336175ee42', u'name': u'zdover'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zdover", "ovpn": "zdover@slim_thinkpad H/78rBqDIJ5AAjMT6sTmKg 49fb166c553d14549491280458269f987bea97295dbf525ffb7c35336175ee42"}, "skip_reason": "Conditional result was False"} 2019-12-21T15:39:27.798 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'rraja@thinkpadp50 8S3Lez75woSBdYiLqQxf2g c9d1666dfc7a3b762700cd250f775a506c397b0f0e6a86289ba555a3546e532b', u'name': u'rraja'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rraja", "ovpn": "rraja@thinkpadp50 8S3Lez75woSBdYiLqQxf2g c9d1666dfc7a3b762700cd250f775a506c397b0f0e6a86289ba555a3546e532b"}, "skip_reason": "Conditional result was False"} 2019-12-21T15:39:27.809 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f', u'name': u'toabctl'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "skip_reason": "Conditional result was False"} 2019-12-21T15:39:27.820 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={u'ovpn': u'mgfritch@laptop 3KhqWETFJaVsH/I8FOBfUg 0aca6c3c6d45c554bad8ba9853f0bc575c9915eee33ee6be2059568ad09a9d72', u'name': u'mgfritch'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 3KhqWETFJaVsH/I8FOBfUg 0aca6c3c6d45c554bad8ba9853f0bc575c9915eee33ee6be2059568ad09a9d72"}, "skip_reason": "Conditional result was False"} 2019-12-21T15:39:27.874 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:39:27 +0000 (0:00:02.109) 0:01:05.881 ***** 2019-12-21T15:39:27.899 INFO:teuthology.task.ansible.out: TASK [users : Filter the revoked_users list] *********************************** skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-21T15:39:27.937 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:39:27 +0000 (0:00:00.064) 0:01:05.945 ***** 2019-12-21T15:39:28.168 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2019-12-21T15:39:28.169 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item=zcerza) => {"ansible_loop_var": "item", "changed": false, "item": "zcerza", "name": "zcerza", "state": "absent"} 2019-12-21T15:39:28.365 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item=aschoen) => {"ansible_loop_var": "item", "changed": false, "item": "aschoen", "name": "aschoen", "state": "absent"} 2019-12-21T15:39:28.560 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item=andrew) => {"ansible_loop_var": "item", "changed": false, "item": "andrew", "name": "andrew", "state": "absent"} 2019-12-21T15:39:28.760 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item=sweil) => {"ansible_loop_var": "item", "changed": false, "item": "sweil", "name": "sweil", "state": "absent"} 2019-12-21T15:39:28.955 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item=brad) => {"ansible_loop_var": "item", "changed": false, "item": "brad", "name": "brad", "state": "absent"} 2019-12-21T15:39:29.151 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item=kefu) => {"ansible_loop_var": "item", "changed": false, "item": "kefu", "name": "kefu", "state": "absent"} 2019-12-21T15:39:29.348 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item=shylesh) => {"ansible_loop_var": "item", "changed": false, "item": "shylesh", "name": "shylesh", "state": "absent"} 2019-12-21T15:39:29.538 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item=trhoden) => {"ansible_loop_var": "item", "changed": false, "item": "trhoden", "name": "trhoden", "state": "absent"} 2019-12-21T15:39:29.738 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item=nishtha) => {"ansible_loop_var": "item", "changed": false, "item": "nishtha", "name": "nishtha", "state": "absent"} 2019-12-21T15:39:29.940 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item=yguang) => {"ansible_loop_var": "item", "changed": false, "item": "yguang", "name": "yguang", "state": "absent"} 2019-12-21T15:39:30.138 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item=sdieffen) => {"ansible_loop_var": "item", "changed": false, "item": "sdieffen", "name": "sdieffen", "state": "absent"} 2019-12-21T15:39:30.334 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item=brian) => {"ansible_loop_var": "item", "changed": false, "item": "brian", "name": "brian", "state": "absent"} 2019-12-21T15:39:30.534 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item=pmcgarry) => {"ansible_loop_var": "item", "changed": false, "item": "pmcgarry", "name": "pmcgarry", "state": "absent"} 2019-12-21T15:39:30.732 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item=karnan) => {"ansible_loop_var": "item", "changed": false, "item": "karnan", "name": "karnan", "state": "absent"} 2019-12-21T15:39:30.932 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item=ryneli) => {"ansible_loop_var": "item", "changed": false, "item": "ryneli", "name": "ryneli", "state": "absent"} 2019-12-21T15:39:31.131 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item=dlambrig) => {"ansible_loop_var": "item", "changed": false, "item": "dlambrig", "name": "dlambrig", "state": "absent"} 2019-12-21T15:39:31.331 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item=icolle) => {"ansible_loop_var": "item", "changed": false, "item": "icolle", "name": "icolle", "state": "absent"} 2019-12-21T15:39:31.527 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item=soumya) => {"ansible_loop_var": "item", "changed": false, "item": "soumya", "name": "soumya", "state": "absent"} 2019-12-21T15:39:31.723 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item=jspray) => {"ansible_loop_var": "item", "changed": false, "item": "jspray", "name": "jspray", "state": "absent"} 2019-12-21T15:39:31.935 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item=loic) => {"ansible_loop_var": "item", "changed": false, "item": "loic", "name": "loic", "state": "absent"} 2019-12-21T15:39:32.132 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item=erwan) => {"ansible_loop_var": "item", "changed": false, "item": "erwan", "name": "erwan", "state": "absent"} 2019-12-21T15:39:32.317 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item=jj) => {"ansible_loop_var": "item", "changed": false, "item": "jj", "name": "jj", "state": "absent"} 2019-12-21T15:39:32.518 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item=amarangone) => {"ansible_loop_var": "item", "changed": false, "item": "amarangone", "name": "amarangone", "state": "absent"} 2019-12-21T15:39:32.713 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item=oprypin) => {"ansible_loop_var": "item", "changed": false, "item": "oprypin", "name": "oprypin", "state": "absent"} 2019-12-21T15:39:32.899 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item=adamyanova) => {"ansible_loop_var": "item", "changed": false, "item": "adamyanova", "name": "adamyanova", "state": "absent"} 2019-12-21T15:39:33.100 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item=sbillah) => {"ansible_loop_var": "item", "changed": false, "item": "sbillah", "name": "sbillah", "state": "absent"} 2019-12-21T15:39:33.300 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item=onyb) => {"ansible_loop_var": "item", "changed": false, "item": "onyb", "name": "onyb", "state": "absent"} 2019-12-21T15:39:33.502 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item=jwilliamson) => {"ansible_loop_var": "item", "changed": false, "item": "jwilliamson", "name": "jwilliamson", "state": "absent"} 2019-12-21T15:39:33.691 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item=kmroz) => {"ansible_loop_var": "item", "changed": false, "item": "kmroz", "name": "kmroz", "state": "absent"} 2019-12-21T15:39:33.878 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item=shehbazj) => {"ansible_loop_var": "item", "changed": false, "item": "shehbazj", "name": "shehbazj", "state": "absent"} 2019-12-21T15:39:34.074 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item=abhishekvrshny) => {"ansible_loop_var": "item", "changed": false, "item": "abhishekvrshny", "name": "abhishekvrshny", "state": "absent"} 2019-12-21T15:39:34.270 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item=asheplyakov) => {"ansible_loop_var": "item", "changed": false, "item": "asheplyakov", "name": "asheplyakov", "state": "absent"} 2019-12-21T15:39:34.465 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item=liupan) => {"ansible_loop_var": "item", "changed": false, "item": "liupan", "name": "liupan", "state": "absent"} 2019-12-21T15:39:34.672 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item=nlevine) => {"ansible_loop_var": "item", "changed": false, "item": "nlevine", "name": "nlevine", "state": "absent"} 2019-12-21T15:39:34.868 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item=tbrekke) => {"ansible_loop_var": "item", "changed": false, "item": "tbrekke", "name": "tbrekke", "state": "absent"} 2019-12-21T15:39:35.055 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item=taco) => {"ansible_loop_var": "item", "changed": false, "item": "taco", "name": "taco", "state": "absent"} 2019-12-21T15:39:35.251 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item=louis) => {"ansible_loop_var": "item", "changed": false, "item": "louis", "name": "louis", "state": "absent"} 2019-12-21T15:39:35.448 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item=amarango) => {"ansible_loop_var": "item", "changed": false, "item": "amarango", "name": "amarango", "state": "absent"} 2019-12-21T15:39:35.636 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item=oobe) => {"ansible_loop_var": "item", "changed": false, "item": "oobe", "name": "oobe", "state": "absent"} 2019-12-21T15:39:35.837 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item=rturk) => {"ansible_loop_var": "item", "changed": false, "item": "rturk", "name": "rturk", "state": "absent"} 2019-12-21T15:39:36.034 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item=fche) => {"ansible_loop_var": "item", "changed": false, "item": "fche", "name": "fche", "state": "absent"} 2019-12-21T15:39:36.101 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:39:36 +0000 (0:00:08.163) 0:01:14.108 ***** 2019-12-21T15:39:36.612 INFO:teuthology.task.ansible.out: TASK [users : Write /keys-repo-sha1 sentinel file] ***************************** 2019-12-21T15:39:36.612 INFO:teuthology.task.ansible.out:changed: [smithi172.front.sepia.ceph.com] => {"changed": true, "checksum": "bcb7fb0987df16cc4ec0eb20567ca03f52e167d8", "dest": "/keys-repo-sha1", "gid": 0, "group": "root", "md5sum": "66384b595a953a298cfd60e4301d14e6", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:etc_runtime_t:s0", "size": 40, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1576942776.13-135918314398727/source", "state": "file", "uid": 0} 2019-12-21T15:39:36.649 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:39:36 +0000 (0:00:00.547) 0:01:14.656 ***** 2019-12-21T15:39:36.674 INFO:teuthology.task.ansible.out: TASK [common : Log the OS name, version and release] *************************** 2019-12-21T15:39:36.676 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => { "msg": "Host smithi172.front.sepia.ceph.com is running RedHat 8.0 (Ootpa)" } 2019-12-21T15:39:36.712 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:39:36 +0000 (0:00:00.063) 0:01:14.720 ***** 2019-12-21T15:39:36.751 INFO:teuthology.task.ansible.out: TASK [common : Including major version specific variables.] ******************** ok: [smithi172.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/redhat_8.yml) => {"ansible_facts": {"epel_repos": {"epel": {"enabled": "{{ enable_epel | ternary(1, 0) }}", "failovermethod": "priority", "gpgcheck": 0, "metalink": "https://mirrors.fedoraproject.org/metalink?repo=epel-$releasever&arch=$basearch&infra=$infra&content=$contentdir", "name": "Extra Packages for Enterprise Linux $releasever - $basearch"}}, "nrpe_selinux_packages": ["python3-libsemanage", "python3-policycoreutils"], "rhsm_repos": ["rhel-8-for-x86_64-baseos-rpms", "rhel-8-for-x86_64-appstream-rpms"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/redhat_8.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/redhat_8.yml"} 2019-12-21T15:39:36.789 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:39:36 +0000 (0:00:00.076) 0:01:14.797 ***** 2019-12-21T15:39:36.817 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 6)] *********************** 2019-12-21T15:39:36.818 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-21T15:39:36.859 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:39:36 +0000 (0:00:00.070) 0:01:14.867 ***** 2019-12-21T15:39:37.132 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 7)] *********************** 2019-12-21T15:39:37.134 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => {"changed": false, "cmd": "timedatectl | grep -E \"Time ?zone\" | sed -e \"s/.*: \\(.*\\) (.*/\\1/\"", "delta": "0:00:00.076760", "end": "2019-12-21 15:39:37.109043", "rc": 0, "start": "2019-12-21 15:39:37.032283", "stderr": "", "stderr_lines": [], "stdout": "Etc/UTC", "stdout_lines": ["Etc/UTC"]} 2019-12-21T15:39:37.170 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:39:37 +0000 (0:00:00.310) 0:01:15.178 ***** 2019-12-21T15:39:37.196 INFO:teuthology.task.ansible.out: TASK [common : Reboot RHEL7 to workaround systemd bug] ************************* 2019-12-21T15:39:37.197 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-21T15:39:37.236 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:39:37 +0000 (0:00:00.065) 0:01:15.243 ***** 2019-12-21T15:39:37.263 INFO:teuthology.task.ansible.out: TASK [common : Wait for reboot in case of systemd workaround] ****************** skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-21T15:39:37.301 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:39:37 +0000 (0:00:00.065) 0:01:15.308 ***** 2019-12-21T15:39:37.327 INFO:teuthology.task.ansible.out: TASK [common : Set /etc/localtime (RHEL/CentOS 6)] ***************************** 2019-12-21T15:39:37.328 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-21T15:39:37.365 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:39:37 +0000 (0:00:00.064) 0:01:15.373 ***** 2019-12-21T15:39:37.391 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone (RHEL/CentOS >= 7)] **************************** 2019-12-21T15:39:37.393 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-21T15:39:37.428 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:39:37 +0000 (0:00:00.063) 0:01:15.436 ***** 2019-12-21T15:39:37.455 INFO:teuthology.task.ansible.out: TASK [common : Install yum on Fedora 22 and later] ***************************** 2019-12-21T15:39:37.456 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-21T15:39:37.497 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:39:37 +0000 (0:00:00.068) 0:01:15.505 ***** 2019-12-21T15:39:37.534 INFO:teuthology.task.ansible.out: TASK [common : Set entitlements_path] ****************************************** 2019-12-21T15:39:37.537 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => {"ansible_facts": {"entitlements_path": "/etc/ansible/secrets/entitlements.yml"}, "changed": false} 2019-12-21T15:39:37.573 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:39:37 +0000 (0:00:00.075) 0:01:15.580 ***** 2019-12-21T15:39:37.619 INFO:teuthology.task.ansible.out: TASK [common : Include Red Hat encrypted variables.] *************************** 2019-12-21T15:39:37.621 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item=None) => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} ok: [smithi172.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2019-12-21T15:39:37.659 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:39:37 +0000 (0:00:00.086) 0:01:15.667 ***** 2019-12-21T15:39:37.707 INFO:teuthology.task.ansible.out: TASK [common : Set have_entitlements] ****************************************** 2019-12-21T15:39:37.709 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => {"ansible_facts": {"have_entitlements": true}, "changed": false} 2019-12-21T15:39:37.744 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:39:37 +0000 (0:00:00.084) 0:01:15.752 ***** 2019-12-21T15:39:38.903 INFO:teuthology.task.ansible.out: TASK [common : Install CA Cert from Satellite Server] ************************** 2019-12-21T15:39:38.905 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": ["Installed /root/.ansible/tmp/ansible-moduletmp-1576942778.784283-saequyo9/katello-ca-consumer-latest.noarch3yd4t3zg.rpm"]} 2019-12-21T15:39:38.946 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:39:38 +0000 (0:00:01.201) 0:01:16.953 ***** 2019-12-21T15:39:39.871 INFO:teuthology.task.ansible.out: TASK [common : Determine if node is registered with subscription-manager.] ***** 2019-12-21T15:39:39.872 INFO:teuthology.task.ansible.out:fatal: [smithi172.front.sepia.ceph.com]: FAILED! => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} ...ignoring 2019-12-21T15:39:39.907 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:39:39 +0000 (0:00:00.961) 0:01:17.915 ***** 2019-12-21T15:39:39.943 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we're already registered] **************** 2019-12-21T15:39:39.945 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => {"ansible_facts": {"rhsm_registered": false}, "changed": false} 2019-12-21T15:39:39.982 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:39:39 +0000 (0:00:00.074) 0:01:17.990 ***** 2019-12-21T15:39:47.459 INFO:teuthology.task.ansible.out: TASK [common : Register with subscription-manager.] **************************** 2019-12-21T15:39:47.461 INFO:teuthology.task.ansible.out:changed: [smithi172.front.sepia.ceph.com] => {"attempts": 1, "censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} 2019-12-21T15:39:47.497 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:39:47 +0000 (0:00:07.515) 0:01:25.505 ***** 2019-12-21T15:39:47.534 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we just registered] ********************** ok: [smithi172.front.sepia.ceph.com] => {"ansible_facts": {"rhsm_registered": true}, "changed": false} 2019-12-21T15:39:47.572 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:39:47 +0000 (0:00:00.074) 0:01:25.580 ***** 2019-12-21T15:39:48.431 INFO:teuthology.task.ansible.out: TASK [common : List CDN releases available to system] ************************** 2019-12-21T15:39:48.433 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => {"changed": false, "cmd": "subscription-manager release --list | grep -E '[0-9]'", "delta": "0:00:00.656615", "end": "2019-12-21 15:39:48.404435", "failed_when_result": false, "rc": 0, "start": "2019-12-21 15:39:47.747820", "stderr": "", "stderr_lines": [], "stdout": "7.4\n7.5\n7.6\n7.7\n7Server\n8\n8.0\n8.1", "stdout_lines": ["7.4", "7.5", "7.6", "7.7", "7Server", "8", "8.0", "8.1"]} 2019-12-21T15:39:48.471 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:39:48 +0000 (0:00:00.898) 0:01:26.478 ***** 2019-12-21T15:39:48.514 INFO:teuthology.task.ansible.out: TASK [common : Unregister Beta/Alpha/RC system with subscription-manager] ****** skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-21T15:39:48.550 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:39:48 +0000 (0:00:00.079) 0:01:26.558 ***** 2019-12-21T15:39:48.576 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered to false if Beta/Alpha/RC release] ********** 2019-12-21T15:39:48.577 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-21T15:39:48.612 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:39:48 +0000 (0:00:00.061) 0:01:26.620 ***** 2019-12-21T15:39:48.638 INFO:teuthology.task.ansible.out: TASK [common : Configure internal beta repos in /etc/yum.repos.d/] ************* 2019-12-21T15:39:48.640 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2019-12-21T15:39:48.677 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:39:48 +0000 (0:00:00.065) 0:01:26.685 ***** 2019-12-21T15:39:48.714 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2019-12-21T15:39:48.715 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-21T15:39:48.753 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:39:48 +0000 (0:00:00.076) 0:01:26.761 ***** 2019-12-21T15:39:55.438 INFO:teuthology.task.ansible.out: TASK [common : Get list of enabled RHSM repos] ********************************* 2019-12-21T15:39:55.439 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => {"changed": false, "cmd": "subscription-manager repos --list | grep -B4 'Enabled:.*1' | grep 'Repo ID:' | sed -e 's/Repo ID:\\s*\\(.*\\)/\\1/' | sort", "delta": "0:00:06.486419", "end": "2019-12-21 15:39:55.419701", "rc": 0, "start": "2019-12-21 15:39:48.933282", "stderr": "", "stderr_lines": [], "stdout": "rhel-8-for-x86_64-appstream-rpms\nrhel-8-for-x86_64-baseos-rpms", "stdout_lines": ["rhel-8-for-x86_64-appstream-rpms", "rhel-8-for-x86_64-baseos-rpms"]} 2019-12-21T15:39:55.478 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:39:55 +0000 (0:00:06.724) 0:01:33.485 ***** 2019-12-21T15:39:55.515 INFO:teuthology.task.ansible.out: TASK [common : Store list of enabled repos] ************************************ 2019-12-21T15:39:55.516 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => {"ansible_facts": {"repo_list": ["rhel-8-for-x86_64-appstream-rpms", "rhel-8-for-x86_64-baseos-rpms"]}, "changed": false} 2019-12-21T15:39:55.553 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:39:55 +0000 (0:00:00.075) 0:01:33.561 ***** 2019-12-21T15:39:55.589 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos false if entitlements are missing or if we unregistered] *** 2019-12-21T15:39:55.591 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-21T15:39:55.629 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:39:55 +0000 (0:00:00.076) 0:01:33.637 ***** 2019-12-21T15:39:55.676 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if rhsm_repos differs from repo_list] **** 2019-12-21T15:39:55.678 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => {"ansible_facts": {"replace_repos": false}, "changed": false} 2019-12-21T15:39:55.716 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:39:55 +0000 (0:00:00.086) 0:01:33.724 ***** 2019-12-21T15:39:55.752 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if newly-subscribed] ********************* 2019-12-21T15:39:55.753 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => {"ansible_facts": {"replace_repos": true}, "changed": false} 2019-12-21T15:39:55.798 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:39:55 +0000 (0:00:00.079) 0:01:33.804 ***** 2019-12-21T15:40:00.209 INFO:teuthology.task.ansible.out: TASK [common : Disable all rhsm repos] ***************************************** 2019-12-21T15:40:00.211 INFO:teuthology.task.ansible.out:changed: [smithi172.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} 2019-12-21T15:40:00.258 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:40:00 +0000 (0:00:04.461) 0:01:38.266 ***** 2019-12-21T15:40:04.110 INFO:teuthology.task.ansible.out: TASK [common : Enable necessary rhsm repos] ************************************ 2019-12-21T15:40:04.111 INFO:teuthology.task.ansible.out:changed: [smithi172.front.sepia.ceph.com] => (item=rhel-8-for-x86_64-baseos-rpms) => {"ansible_loop_var": "item", "changed": true, "cmd": ["subscription-manager", "repos", "--enable", "rhel-8-for-x86_64-baseos-rpms"], "delta": "0:00:03.612167", "end": "2019-12-21 15:40:04.082301", "item": "rhel-8-for-x86_64-baseos-rpms", "rc": 0, "start": "2019-12-21 15:40:00.470134", "stderr": "", "stderr_lines": [], "stdout": "Repository 'rhel-8-for-x86_64-baseos-rpms' is enabled for this system.", "stdout_lines": ["Repository 'rhel-8-for-x86_64-baseos-rpms' is enabled for this system."]} 2019-12-21T15:40:08.121 INFO:teuthology.task.ansible.out:changed: [smithi172.front.sepia.ceph.com] => (item=rhel-8-for-x86_64-appstream-rpms) => {"ansible_loop_var": "item", "changed": true, "cmd": ["subscription-manager", "repos", "--enable", "rhel-8-for-x86_64-appstream-rpms"], "delta": "0:00:03.824414", "end": "2019-12-21 15:40:08.097241", "item": "rhel-8-for-x86_64-appstream-rpms", "rc": 0, "start": "2019-12-21 15:40:04.272827", "stderr": "", "stderr_lines": [], "stdout": "Repository 'rhel-8-for-x86_64-appstream-rpms' is enabled for this system.", "stdout_lines": ["Repository 'rhel-8-for-x86_64-appstream-rpms' is enabled for this system."]} 2019-12-21T15:40:08.162 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:40:08 +0000 (0:00:07.904) 0:01:46.170 ***** 2019-12-21T15:40:08.633 INFO:teuthology.task.ansible.out: TASK [common : Remove old apt-mirror repository definition.] ******************* ok: [smithi172.front.sepia.ceph.com] => {"changed": false, "path": "/etc/yum.repos.d/cd.repo", "state": "absent"} 2019-12-21T15:40:08.669 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:40:08 +0000 (0:00:00.506) 0:01:46.677 ***** 2019-12-21T15:40:08.874 INFO:teuthology.task.ansible.out: TASK [common : Increase the yum timeout.] ************************************** ok: [smithi172.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2019-12-21T15:40:08.911 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:40:08 +0000 (0:00:00.241) 0:01:46.919 ***** 2019-12-21T15:40:09.424 INFO:teuthology.task.ansible.out: TASK [common : Configure epel repos in /etc/yum.repos.d/] ********************** 2019-12-21T15:40:09.425 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'value': {u'gpgcheck': 0, u'metalink': u'https://mirrors.fedoraproject.org/metalink?repo=epel-$releasever&arch=$basearch&infra=$infra&content=$contentdir', u'enabled': u'1', u'name': u'Extra Packages for Enterprise Linux $releasever - $basearch', u'failovermethod': u'priority'}, 'key': u'epel'}) => {"ansible_loop_var": "item", "changed": false, "checksum": "e6ba489657e79a66ebaff7f2cc203b7a42394494", "dest": "/etc/yum.repos.d/epel.repo", "gid": 0, "group": "root", "item": {"key": "epel", "value": {"enabled": "1", "failovermethod": "priority", "gpgcheck": 0, "metalink": "https://mirrors.fedoraproject.org/metalink?repo=epel-$releasever&arch=$basearch&infra=$infra&content=$contentdir", "name": "Extra Packages for Enterprise Linux $releasever - $basearch"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 331, "state": "file", "uid": 0} 2019-12-21T15:40:09.464 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:40:09 +0000 (0:00:00.552) 0:01:47.471 ***** 2019-12-21T15:40:10.039 INFO:teuthology.task.ansible.out: TASK [common : Configure local epel mirrorlists] ******************************* 2019-12-21T15:40:10.041 INFO:teuthology.task.ansible.out:changed: [smithi172.front.sepia.ceph.com] => (item=epel-mirrorlist) => {"ansible_loop_var": "item", "changed": true, "checksum": "3c9ba7bacd1e30bed014cefd1fb4eab0f0f8fa10", "dest": "/etc/yum.repos.d/epel-mirrorlist", "gid": 0, "group": "root", "item": "epel-mirrorlist", "md5sum": "c0b17108eae19ce4e019023f4d971436", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:system_conf_t:s0", "size": 484, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1576942809.5-17630642768761/source", "state": "file", "uid": 0} 2019-12-21T15:40:10.563 INFO:teuthology.task.ansible.out:changed: [smithi172.front.sepia.ceph.com] => (item=epel-testing-mirrorlist) => {"ansible_loop_var": "item", "changed": true, "checksum": "d542c3e2c30a0d3eb7d454775342c302c0ded15b", "dest": "/etc/yum.repos.d/epel-testing-mirrorlist", "gid": 0, "group": "root", "item": "epel-testing-mirrorlist", "md5sum": "8ea2607820b97348115878813225ac00", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:system_conf_t:s0", "size": 540, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1576942810.05-145475453963957/source", "state": "file", "uid": 0} 2019-12-21T15:40:10.609 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:40:10 +0000 (0:00:01.145) 0:01:48.616 ***** 2019-12-21T15:40:11.113 INFO:teuthology.task.ansible.out: TASK [common : Temporarily comment mirrorlist so we can update yum cache] ****** 2019-12-21T15:40:11.114 INFO:teuthology.task.ansible.out:changed: [smithi172.front.sepia.ceph.com] => (item={'value': {u'gpgcheck': 0, u'metalink': u'https://mirrors.fedoraproject.org/metalink?repo=epel-$releasever&arch=$basearch&infra=$infra&content=$contentdir', u'enabled': u'1', u'name': u'Extra Packages for Enterprise Linux $releasever - $basearch', u'failovermethod': u'priority'}, 'key': u'epel'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "epel", "value": {"enabled": "1", "failovermethod": "priority", "gpgcheck": 0, "metalink": "https://mirrors.fedoraproject.org/metalink?repo=epel-$releasever&arch=$basearch&infra=$infra&content=$contentdir", "name": "Extra Packages for Enterprise Linux $releasever - $basearch"}}, "msg": "1 replacements made"} 2019-12-21T15:40:11.155 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:40:11 +0000 (0:00:00.545) 0:01:49.162 ***** 2019-12-21T15:40:16.186 INFO:teuthology.task.ansible.out: TASK [common : Update yum cache] *********************************************** 2019-12-21T15:40:16.191 INFO:teuthology.task.ansible.out: [WARNING]: Consider using the file module with state=absent rather than running 'rm'. If you need to use command because file is insufficient you can add 'warn: false' to this command task or set 'command_warnings=False' in ansible.cfg to get rid of this message. changed: [smithi172.front.sepia.ceph.com] => {"changed": true, "cmd": "rm -rf /var/cache/yum/*; yum clean all; yum makecache", "delta": "0:00:04.825284", "end": "2019-12-21 15:40:16.167098", "rc": 0, "start": "2019-12-21 15:40:11.341814", "stderr": "Cannot find a valid baseurl for repo: epel, ignoring this repo.", "stderr_lines": ["Cannot find a valid baseurl for repo: epel, ignoring this repo."], "stdout": "Updating Subscription Management repositories.\n195 files removed\nUpdating Subscription Management repositories.\nRed Hat Enterprise Linux 8 for x86_64 - AppStre 62 MB/s | 9.4 MB 00:00 \nRed Hat Enterprise Linux 8 for x86_64 - BaseOS 57 MB/s | 8.1 MB 00:00 \nLast metadata expiration check: 0:00:01 ago on Sat 21 Dec 2019 03:40:14 PM UTC.\nMetadata cache created.", "stdout_lines": ["Updating Subscription Management repositories.", "195 files removed", "Updating Subscription Management repositories.", "Red Hat Enterprise Linux 8 for x86_64 - AppStre 62 MB/s | 9.4 MB 00:00 ", "Red Hat Enterprise Linux 8 for x86_64 - BaseOS 57 MB/s | 8.1 MB 00:00 ", "Last metadata expiration check: 0:00:01 ago on Sat 21 Dec 2019 03:40:14 PM UTC.", "Metadata cache created."]} 2019-12-21T15:40:16.232 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:40:16 +0000 (0:00:05.077) 0:01:54.240 ***** 2019-12-21T15:40:16.446 INFO:teuthology.task.ansible.out: TASK [common : Uncomment mirrorlist] ******************************************* 2019-12-21T15:40:16.447 INFO:teuthology.task.ansible.out:changed: [smithi172.front.sepia.ceph.com] => (item={'value': {u'gpgcheck': 0, u'metalink': u'https://mirrors.fedoraproject.org/metalink?repo=epel-$releasever&arch=$basearch&infra=$infra&content=$contentdir', u'enabled': u'1', u'name': u'Extra Packages for Enterprise Linux $releasever - $basearch', u'failovermethod': u'priority'}, 'key': u'epel'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "epel", "value": {"enabled": "1", "failovermethod": "priority", "gpgcheck": 0, "metalink": "https://mirrors.fedoraproject.org/metalink?repo=epel-$releasever&arch=$basearch&infra=$infra&content=$contentdir", "name": "Extra Packages for Enterprise Linux $releasever - $basearch"}}, "msg": "1 replacements made"} 2019-12-21T15:40:16.500 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:40:16 +0000 (0:00:00.268) 0:01:54.508 ***** 2019-12-21T15:40:16.526 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2019-12-21T15:40:16.527 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-21T15:40:16.564 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:40:16 +0000 (0:00:00.064) 0:01:54.572 ***** 2019-12-21T15:40:16.591 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone in /etc/timezone] ****************************** skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-21T15:40:16.636 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:40:16 +0000 (0:00:00.071) 0:01:54.644 ***** 2019-12-21T15:40:16.662 INFO:teuthology.task.ansible.out: TASK [common : Inform the OS of the new timezone] ****************************** 2019-12-21T15:40:16.664 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-21T15:40:16.699 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:40:16 +0000 (0:00:00.062) 0:01:54.707 ***** 2019-12-21T15:40:16.733 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-21T15:40:16.764 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:40:16 +0000 (0:00:00.065) 0:01:54.772 ***** 2019-12-21T15:40:16.790 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone] *********************************************** 2019-12-21T15:40:16.792 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-21T15:40:16.835 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:40:16 +0000 (0:00:00.070) 0:01:54.842 ***** 2019-12-21T15:40:16.860 INFO:teuthology.task.ansible.out: TASK [common : Add base OpenSUSE Leap repo] ************************************ 2019-12-21T15:40:16.862 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-21T15:40:16.901 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:40:16 +0000 (0:00:00.066) 0:01:54.909 ***** 2019-12-21T15:40:16.932 INFO:teuthology.task.ansible.out: TASK [common : Add updates OpenSUSE Leap repo] ********************************* 2019-12-21T15:40:16.934 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-21T15:40:16.975 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:40:16 +0000 (0:00:00.073) 0:01:54.982 ***** 2019-12-21T15:40:17.001 INFO:teuthology.task.ansible.out: TASK [common : Refresh repos] ************************************************** 2019-12-21T15:40:17.003 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-21T15:40:17.044 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:40:17 +0000 (0:00:00.069) 0:01:55.051 ***** 2019-12-21T15:40:17.524 INFO:teuthology.task.ansible.out: TASK [common : Set the hardware clock] ***************************************** 2019-12-21T15:40:17.526 INFO:teuthology.task.ansible.out:changed: [smithi172.front.sepia.ceph.com] => {"changed": true, "cmd": ["hwclock", "--systohc"], "delta": "0:00:00.289490", "end": "2019-12-21 15:40:17.500391", "rc": 0, "start": "2019-12-21 15:40:17.210901", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2019-12-21T15:40:17.566 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:40:17 +0000 (0:00:00.521) 0:01:55.573 ***** 2019-12-21T15:40:20.696 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (RedHat)] ***************************** 2019-12-21T15:40:20.696 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": ["Installed: krb5-workstation"]} 2019-12-21T15:40:20.736 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:40:20 +0000 (0:00:03.170) 0:01:58.743 ***** 2019-12-21T15:40:20.765 INFO:teuthology.task.ansible.out: TASK [common : Clean apt cache] ************************************************ 2019-12-21T15:40:20.766 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-21T15:40:20.812 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:40:20 +0000 (0:00:00.072) 0:01:58.816 ***** 2019-12-21T15:40:20.843 INFO:teuthology.task.ansible.out: TASK [common : Update apt cache] *********************************************** 2019-12-21T15:40:20.844 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-21T15:40:20.884 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:40:20 +0000 (0:00:00.074) 0:01:58.891 ***** 2019-12-21T15:40:20.921 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (Debian)] ***************************** skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-21T15:40:20.960 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:40:20 +0000 (0:00:00.076) 0:01:58.967 ***** 2019-12-21T15:40:20.989 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (OpenSUSE Leap)] ********************** 2019-12-21T15:40:20.990 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-21T15:40:21.028 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:40:21 +0000 (0:00:00.068) 0:01:59.036 ***** 2019-12-21T15:40:21.499 INFO:teuthology.task.ansible.out: TASK [common : Add krb5 config file] ******************************************* 2019-12-21T15:40:21.500 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => {"changed": false, "checksum": "b3c1a33d19a4e448c29374429629dff57657b576", "dest": "/etc/krb5.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/krb5.conf", "secontext": "system_u:object_r:krb5_conf_t:s0", "size": 131, "state": "file", "uid": 0} 2019-12-21T15:40:21.569 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:40:21 +0000 (0:00:00.537) 0:01:59.573 ***** 2019-12-21T15:40:23.131 INFO:teuthology.task.ansible.out: TASK [common : Make sure smartmontools is installed] *************************** 2019-12-21T15:40:23.132 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": ["Installed: smartmontools"]} 2019-12-21T15:40:23.187 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:40:23 +0000 (0:00:01.620) 0:02:01.194 ***** 2019-12-21T15:40:23.886 INFO:teuthology.task.ansible.out: TASK [common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/.] *** 2019-12-21T15:40:23.888 INFO:teuthology.task.ansible.out:changed: [smithi172.front.sepia.ceph.com] => (item=megacli) => {"ansible_loop_var": "item", "changed": true, "checksum": "cb3ef2aa6a3e3002b234f166e739a39b1f3a1bde", "dest": "/usr/sbin/megacli", "gid": 0, "group": "root", "item": "megacli", "md5sum": "da0dc6d43b293333920cda2d1fc220e2", "mode": "0755", "owner": "root", "secontext": "system_u:object_r:bin_t:s0", "size": 2716736, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1576942823.21-272680567188508/source", "state": "file", "uid": 0} 2019-12-21T15:40:24.549 INFO:teuthology.task.ansible.out:changed: [smithi172.front.sepia.ceph.com] => (item=cli64) => {"ansible_loop_var": "item", "changed": true, "checksum": "a3b8773b14e8594ffbd9fc156c09436eb0184bc7", "dest": "/usr/sbin/cli64", "gid": 0, "group": "root", "item": "cli64", "md5sum": "5e1d5cbeb339bbe2906752233e88486d", "mode": "0755", "owner": "root", "secontext": "system_u:object_r:bin_t:s0", "size": 1952038, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1576942823.89-270498891895219/source", "state": "file", "uid": 0} 2019-12-21T15:40:25.139 INFO:teuthology.task.ansible.out:changed: [smithi172.front.sepia.ceph.com] => (item=nvme) => {"ansible_loop_var": "item", "changed": true, "checksum": "98735dcacd6fec55dd92b67281b478603b7955b1", "dest": "/usr/sbin/nvme", "gid": 0, "group": "root", "item": "nvme", "md5sum": "f8f4b724d9a4fc98a48d3ba3cb2748e2", "mode": "0755", "owner": "root", "secontext": "system_u:object_r:bin_t:s0", "size": 580432, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1576942824.56-42919280926929/source", "state": "file", "uid": 0} 2019-12-21T15:40:25.192 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:40:25 +0000 (0:00:02.002) 0:02:03.197 ***** 2019-12-21T15:40:25.393 INFO:teuthology.task.ansible.out: TASK [common : Create /usr/libexec.] ******************************************* 2019-12-21T15:40:25.395 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/libexec", "secontext": "system_u:object_r:bin_t:s0", "size": 4096, "state": "directory", "uid": 0} 2019-12-21T15:40:25.445 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:40:25 +0000 (0:00:00.255) 0:02:03.452 ***** 2019-12-21T15:40:25.986 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** 2019-12-21T15:40:25.987 INFO:teuthology.task.ansible.out:changed: [smithi172.front.sepia.ceph.com] => (item=smart.sh) => {"ansible_loop_var": "item", "changed": true, "checksum": "9a447e184389aadb9aaf1e77f30f0a7f8b716c4e", "dest": "/usr/libexec/smart.sh", "gid": 0, "group": "root", "item": "smart.sh", "md5sum": "a1185825c55c0301955d1c6f48afa459", "mode": "0755", "owner": "root", "secontext": "system_u:object_r:bin_t:s0", "size": 12725, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1576942825.47-123288431446014/source", "state": "file", "uid": 0} 2019-12-21T15:40:26.542 INFO:teuthology.task.ansible.out:changed: [smithi172.front.sepia.ceph.com] => (item=raid.pl) => {"ansible_loop_var": "item", "changed": true, "checksum": "9d9a89805d8a9e4c7281934e887da31f7ac464fb", "dest": "/usr/libexec/raid.pl", "gid": 0, "group": "root", "item": "raid.pl", "md5sum": "2ff1d0b7420a46c24916a3f7cba75004", "mode": "0755", "owner": "root", "secontext": "system_u:object_r:bin_t:s0", "size": 8602, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1576942825.99-126434364420723/source", "state": "file", "uid": 0} 2019-12-21T15:40:27.046 INFO:teuthology.task.ansible.out:changed: [smithi172.front.sepia.ceph.com] => (item=diskusage.pl) => {"ansible_loop_var": "item", "changed": true, "checksum": "9c74425885a83f23bc37817de46979ca5173ccb7", "dest": "/usr/libexec/diskusage.pl", "gid": 0, "group": "root", "item": "diskusage.pl", "md5sum": "a826600dbfe6a128fb60ddc413ef1069", "mode": "0755", "owner": "root", "secontext": "system_u:object_r:bin_t:s0", "size": 3605, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1576942826.54-18814059564250/source", "state": "file", "uid": 0} 2019-12-21T15:40:27.100 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:40:27 +0000 (0:00:01.655) 0:02:05.108 ***** 2019-12-21T15:40:27.136 INFO:teuthology.task.ansible.out: TASK [common : Include apt_system vars] **************************************** 2019-12-21T15:40:27.137 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-21T15:40:27.181 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:40:27 +0000 (0:00:00.080) 0:02:05.188 ***** 2019-12-21T15:40:27.227 INFO:teuthology.task.ansible.out: TASK [common : Include yum_systems vars] *************************************** 2019-12-21T15:40:27.229 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => {"ansible_facts": {"nagios_plugins_directory": "/usr/lib64/nagios/plugins", "nrpe_group": "nrpe", "nrpe_packages": ["nagios-common", "nrpe", "nagios-plugins", "nagios-plugins-load"], "nrpe_service_name": "nrpe", "nrpe_user": "nrpe"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/yum_systems.yml"], "changed": false} 2019-12-21T15:40:27.268 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:40:27 +0000 (0:00:00.087) 0:02:05.276 ***** 2019-12-21T15:40:27.471 INFO:teuthology.task.ansible.out: TASK [common : Check for epel] ************************************************* 2019-12-21T15:40:27.473 INFO:teuthology.task.ansible.out:changed: [smithi172.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} 2019-12-21T15:40:27.508 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:40:27 +0000 (0:00:00.239) 0:02:05.516 ***** 2019-12-21T15:40:27.555 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe without epel] ************************************** 2019-12-21T15:40:27.557 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2019-12-21T15:40:27.567 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2019-12-21T15:40:27.579 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2019-12-21T15:40:27.591 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2019-12-21T15:40:27.637 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:40:27 +0000 (0:00:00.127) 0:02:05.644 ***** 2019-12-21T15:40:36.550 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (RHEL/CentOS)] ************ 2019-12-21T15:40:36.551 INFO:teuthology.task.ansible.out:changed: [smithi172.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Installed: nagios-common", "Installed: nrpe", "Installed: nagios-plugins", "Installed: nagios-plugins-load", "Installed: nrpe-3.2.1-9.el8.x86_64", "Installed: nagios-common-4.4.5-1.el8.x86_64", "Installed: nagios-plugins-2.2.2-2.20190926git1b8ad57.el8.x86_64", "Installed: nagios-plugins-load-2.2.2-2.20190926git1b8ad57.el8.x86_64"]} 2019-12-21T15:40:36.595 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:40:36 +0000 (0:00:08.958) 0:02:14.602 ***** 2019-12-21T15:40:36.632 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (Ubuntu)] ***************** 2019-12-21T15:40:36.634 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-21T15:40:36.672 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:40:36 +0000 (0:00:00.075) 0:02:14.677 ***** 2019-12-21T15:40:37.236 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios sudoers.d for raid utilities.] ******************** 2019-12-21T15:40:37.238 INFO:teuthology.task.ansible.out:changed: [smithi172.front.sepia.ceph.com] => {"changed": true, "checksum": "b423b985ccbb69b65a711d372645a5f704b30e8b", "dest": "/etc/sudoers.d/90-nagios", "gid": 0, "group": "root", "md5sum": "acdfbe59732074480b7f015fa152a14e", "mode": "0440", "owner": "root", "secontext": "system_u:object_r:etc_t:s0", "size": 178, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1576942836.69-237019914623709/source", "state": "file", "uid": 0} 2019-12-21T15:40:37.273 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:40:37 +0000 (0:00:00.603) 0:02:15.281 ***** 2019-12-21T15:40:37.813 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios check_mem script] ********************************* 2019-12-21T15:40:37.814 INFO:teuthology.task.ansible.out:changed: [smithi172.front.sepia.ceph.com] => {"changed": true, "checksum": "39df629b10ded370443e2e4c84d690332c95104d", "dest": "/usr/lib64/nagios/plugins/check_mem.sh", "gid": 0, "group": "root", "md5sum": "1f0ee88f0225fe8e005cd4f9fbc5e2d9", "mode": "0755", "owner": "root", "secontext": "system_u:object_r:nagios_unconfined_plugin_exec_t:s0", "size": 1964, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1576942837.31-100390441322437/source", "state": "file", "uid": 0} 2019-12-21T15:40:37.857 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:40:37 +0000 (0:00:00.583) 0:02:15.864 ***** 2019-12-21T15:40:37.892 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (Ubuntu)] ************************ 2019-12-21T15:40:37.893 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-21T15:40:37.938 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:40:37 +0000 (0:00:00.077) 0:02:15.942 ***** 2019-12-21T15:40:38.138 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (RHEL/CentOS)] ******************* 2019-12-21T15:40:38.139 INFO:teuthology.task.ansible.out:changed: [smithi172.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line replaced"} 2019-12-21T15:40:38.182 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:40:38 +0000 (0:00:00.245) 0:02:16.188 ***** 2019-12-21T15:40:38.435 INFO:teuthology.task.ansible.out: TASK [common : Check firewalld status] ***************************************** 2019-12-21T15:40:38.436 INFO:teuthology.task.ansible.out:fatal: [smithi172.front.sepia.ceph.com]: FAILED! => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} ...ignoring 2019-12-21T15:40:38.474 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:40:38 +0000 (0:00:00.293) 0:02:16.481 ***** 2019-12-21T15:40:38.521 INFO:teuthology.task.ansible.out: TASK [common : Open nrpe port if firewalld enabled] **************************** 2019-12-21T15:40:38.522 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-21T15:40:38.560 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:40:38 +0000 (0:00:00.086) 0:02:16.567 ***** 2019-12-21T15:40:39.121 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios nrpe config.] ************************************* 2019-12-21T15:40:39.122 INFO:teuthology.task.ansible.out:changed: [smithi172.front.sepia.ceph.com] => {"changed": true, "checksum": "333141ba5e85aea94f51511a52d9b115d2e0112f", "dest": "/etc/nagios/nrpe.cfg", "gid": 0, "group": "root", "md5sum": "a2f2d08aa8aa3493038c91c0fd4b7279", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:nrpe_etc_t:s0", "size": 1174, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1576942838.58-174222066431590/source", "state": "file", "uid": 0} 2019-12-21T15:40:39.163 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:40:39 +0000 (0:00:00.602) 0:02:17.170 ***** 2019-12-21T15:40:40.032 INFO:teuthology.task.ansible.out: TASK [common : Make sure nagios nrpe service is running.] ********************** 2019-12-21T15:40:40.034 INFO:teuthology.task.ansible.out:changed: [smithi172.front.sepia.ceph.com] => {"changed": true, "enabled": true, "name": "nrpe", "state": "started", "status": {"ActiveEnterTimestampMonotonic": "0", "ActiveExitTimestampMonotonic": "0", "ActiveState": "inactive", "After": "nss-lookup.target local-fs.target sysinit.target system.slice -.mount tmp.mount network.target time-sync.target basic.target systemd-journald.socket var-run.mount systemd-tmpfiles-setup.service", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "no", "AssertTimestampMonotonic": "0", "Before": "shutdown.target plymouth-quit.service getty@tty1.service xdm.service", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "no", "ConditionTimestampMonotonic": "0", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.target nrpe.socket", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "Nagios Remote Plugin Executor", "DevicePolicy": "auto", "Documentation": "http://www.nagios.org/documentation", "DynamicUser": "no", "ExecMainCode": "0", "ExecMainExitTimestampM 2019-12-21T15:40:40.035 INFO:teuthology.task.ansible.out:onotonic": "0", "ExecMainPID": "0", "ExecMainStartTimestampMonotonic": "0", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/nrpe ; argv[]=/usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -f ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStopPost": "{ path=/bin/rm ; argv[]=/bin/rm -f /var/run/nrpe/nrpe.pid ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/nrpe.service", "GID": "[not set]", "Group": "nrpe", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "nrpe.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestampMonotonic": "0", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "infinity", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "16777216", "LimitMEMLOCKSoft": "16777216", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "127463", "LimitNPROCSoft": "127463", "LimitRSS": "infinity 2019-12-21T15:40:40.037 INFO:teuthology.task.ansible.out:", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127463", "LimitSIGPENDINGSoft": "127463", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "0", "MemoryAccounting": "yes", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "nrpe.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "-500", "OnFailureJobMode": "replace", "PIDFile": "/var/run/nrpe/nrpe.pid", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateMounts": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "system.slice sysinit.target -.mount", "RequiresMountsFor": "/run/nrpe /var/tmp", "Restart": "on-abort", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectory": "nrpe", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", 2019-12-21T15:40:40.038 INFO:teuthology.task.ansible.out:"StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestampMonotonic": "0", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "dead", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "[not set]", "TasksMax": "26213", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min", "TimerSlackNSec": "50000", "Transient": "no", "Type": "simple", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "disabled", "UnitFileState": "disabled", "User": "nrpe", "UtmpMode": "init", "WatchdogTimestampMonotonic": "0", "WatchdogUSec": "0"}} 2019-12-21T15:40:40.078 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:40:40 +0000 (0:00:00.915) 0:02:18.085 ***** 2019-12-21T15:40:40.274 INFO:teuthology.task.ansible.out: TASK [common : Get SELinux status] ********************************************* 2019-12-21T15:40:40.276 INFO:teuthology.task.ansible.out:changed: [smithi172.front.sepia.ceph.com] => {"changed": true, "cmd": ["getenforce"], "delta": "0:00:00.001664", "end": "2019-12-21 15:40:40.254601", "rc": 0, "start": "2019-12-21 15:40:40.252937", "stderr": "", "stderr_lines": [], "stdout": "Permissive", "stdout_lines": ["Permissive"]} 2019-12-21T15:40:40.327 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:40:40 +0000 (0:00:00.248) 0:02:18.334 ***** 2019-12-21T15:40:42.028 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Install SELinux dependencies] **************************** 2019-12-21T15:40:42.030 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": ["Installed: python3-libsemanage", "Installed: python3-policycoreutils"]} 2019-12-21T15:40:42.074 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:40:42 +0000 (0:00:01.747) 0:02:20.081 ***** 2019-12-21T15:40:43.787 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Ensure SELinux policy is up to date] ********************* 2019-12-21T15:40:43.791 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": ["Installed: selinux-policy-targeted"]} 2019-12-21T15:40:43.831 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:40:43 +0000 (0:00:01.756) 0:02:21.838 ***** 2019-12-21T15:40:45.940 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Set SELinux boolean nagios_run_sudo true] **************** 2019-12-21T15:40:45.941 INFO:teuthology.task.ansible.out:changed: [smithi172.front.sepia.ceph.com] => {"changed": true, "name": "nagios_run_sudo", "persistent": true, "state": true} 2019-12-21T15:40:45.981 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:40:45 +0000 (0:00:02.150) 0:02:23.989 ***** 2019-12-21T15:40:46.288 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove SELinux policy package] *************************** 2019-12-21T15:40:46.289 INFO:teuthology.task.ansible.out:changed: [smithi172.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule", "-r", "nrpe"], "delta": "0:00:00.106425", "end": "2019-12-21 15:40:46.262890", "failed_when_result": false, "msg": "non-zero return code", "rc": 1, "start": "2019-12-21 15:40:46.156465", "stderr": "libsemanage.semanage_direct_remove_key: Unable to remove module nrpe at priority 400. (No such file or directory).\nsemodule: Failed!", "stderr_lines": ["libsemanage.semanage_direct_remove_key: Unable to remove module nrpe at priority 400. (No such file or directory).", "semodule: Failed!"], "stdout": "", "stdout_lines": []} 2019-12-21T15:40:46.327 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:40:46 +0000 (0:00:00.344) 0:02:24.333 ***** 2019-12-21T15:40:46.851 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Copy SELinux type enforcement file] ********************** changed: [smithi172.front.sepia.ceph.com] => {"changed": true, "checksum": "99d9b2ea41fb2d6613898129718064833d2de428", "dest": "/tmp/nrpe.te", "gid": 0, "group": "root", "md5sum": "286c682fc73965c17876891a4581da3e", "mode": "0644", "owner": "root", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 934, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1576942846.35-278396629171000/source", "state": "file", "uid": 0} 2019-12-21T15:40:46.887 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:40:46 +0000 (0:00:00.561) 0:02:24.895 ***** 2019-12-21T15:40:47.116 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Compile SELinux module file] ***************************** 2019-12-21T15:40:47.117 INFO:teuthology.task.ansible.out:changed: [smithi172.front.sepia.ceph.com] => {"changed": true, "cmd": ["checkmodule", "-M", "-m", "-o", "/tmp/nrpe.mod", "/tmp/nrpe.te"], "delta": "0:00:00.030942", "end": "2019-12-21 15:40:47.096565", "rc": 0, "start": "2019-12-21 15:40:47.065623", "stderr": "", "stderr_lines": [], "stdout": "checkmodule: loading policy configuration from /tmp/nrpe.te\ncheckmodule: policy configuration loaded\ncheckmodule: writing binary representation (version 19) to /tmp/nrpe.mod", "stdout_lines": ["checkmodule: loading policy configuration from /tmp/nrpe.te", "checkmodule: policy configuration loaded", "checkmodule: writing binary representation (version 19) to /tmp/nrpe.mod"]} 2019-12-21T15:40:47.172 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:40:47 +0000 (0:00:00.284) 0:02:25.180 ***** 2019-12-21T15:40:47.395 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Build SELinux policy package] **************************** 2019-12-21T15:40:47.396 INFO:teuthology.task.ansible.out:changed: [smithi172.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule_package", "-o", "/tmp/nrpe.pp", "-m", "/tmp/nrpe.mod"], "delta": "0:00:00.013394", "end": "2019-12-21 15:40:47.363287", "rc": 0, "start": "2019-12-21 15:40:47.349893", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2019-12-21T15:40:47.446 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:40:47 +0000 (0:00:00.273) 0:02:25.453 ***** 2019-12-21T15:41:03.524 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Load SELinux policy package] ***************************** changed: [smithi172.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule", "-i", "/tmp/nrpe.pp"], "delta": "0:00:15.878549", "end": "2019-12-21 15:41:03.500634", "rc": 0, "start": "2019-12-21 15:40:47.622085", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2019-12-21T15:41:03.561 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:41:03 +0000 (0:00:16.115) 0:02:41.569 ***** 2019-12-21T15:41:03.767 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove temporary files] ********************************** ok: [smithi172.front.sepia.ceph.com] => {"changed": false, "path": "/tmp/nrpe.*", "state": "absent"} 2019-12-21T15:41:03.801 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:41:03 +0000 (0:00:00.239) 0:02:41.809 ***** 2019-12-21T15:41:03.827 INFO:teuthology.task.ansible.out: TASK [testnode : Include package type specific vars.] ************************** 2019-12-21T15:41:03.828 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => {"ansible_facts": {"ceph_dependency_packages_to_remove": ["boost-random", "boost-program-options", "leveldb", "xmlstarlet", "python-jinja2", "python-ceph", "python-flask", "python-requests", "boost-random", "python-urllib3", "python-babel", "hdparm", "python-markupsafe", "python-werkzeug", "python-itsdangerous"], "ceph_packages_to_remove": ["ceph", "ceph-base", "ceph-selinux", "ceph-common", "ceph-debuginfo", "ceph-release", "libcephfs1", "ceph-radosgw", "python-ceph", "python-rados", "python-rbd", "python-cephfs", "librbd1", "librados2", "mod_fastcgi", "iozone"], "ntp_service_name": "ntpd", "packages_to_remove": ["device-mapper-multipath"], "pip_packages_to_install": ["remoto>=0.0.35"], "ssh_service_name": "sshd"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/dnf_systems.yml"], "changed": false} 2019-12-21T15:41:03.869 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:41:03 +0000 (0:00:00.068) 0:02:41.877 ***** 2019-12-21T15:41:03.916 INFO:teuthology.task.ansible.out: TASK [testnode : Including distro specific variables.] ************************* 2019-12-21T15:41:03.917 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2019-12-21T15:41:03.959 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:41:03 +0000 (0:00:00.089) 0:02:41.966 ***** 2019-12-21T15:41:03.995 INFO:teuthology.task.ansible.out: TASK [testnode : Including major version specific variables.] ****************** 2019-12-21T15:41:03.996 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/redhat_8.yml) => {"ansible_facts": {"common_yum_repos": {"lab-extras": {"baseurl": "http://{{ mirror_host }}/lab-extras/8/", "enabled": 1, "gpgcheck": 0, "name": "lab-extras"}}, "copr_repos": ["ktdreyer/ceph-el8"], "epel_packages": [], "nfs_service": "nfs-server", "ntp_service_name": "chronyd", "packages": ["@core", "@base", "dnf-utils", "git-all", "sysstat", "libedit", "boost-thread", "xfsprogs", "gdisk", "parted", "libgcrypt", "fuse-libs", "openssl", "libuuid", "attr", "ant", "lsof", "gettext", "bc", "xfsdump", "blktrace", "usbredir", "podman", "libev-devel", "ncurses-devel", "gcc", "git", "python3-nose", "python3-virtualenv", "genisoimage", "qemu-img", "libacl-devel", "dbench", "autoconf"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/redhat_8.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/redhat_8.yml"} 2019-12-21T15:41:04.037 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:41:04 +0000 (0:00:00.078) 0:02:42.045 ***** 2019-12-21T15:41:04.074 INFO:teuthology.task.ansible.out: TASK [testnode : Including version specific variables.] ************************ 2019-12-21T15:41:04.075 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2019-12-21T15:41:04.112 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:41:04 +0000 (0:00:00.075) 0:02:42.120 ***** 2019-12-21T15:41:04.304 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the sudo group exists.] ******************************** ok: [smithi172.front.sepia.ceph.com] => {"changed": false, "gid": 1000, "name": "sudo", "state": "present", "system": false} 2019-12-21T15:41:04.342 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:41:04 +0000 (0:00:00.229) 0:02:42.349 ***** 2019-12-21T15:41:04.535 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the teuthology_user group exists.] ********************* ok: [smithi172.front.sepia.ceph.com] => {"changed": false, "gid": 1205, "name": "ubuntu", "state": "present", "system": false} 2019-12-21T15:41:04.576 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:41:04 +0000 (0:00:00.233) 0:02:42.583 ***** 2019-12-21T15:41:04.602 INFO:teuthology.task.ansible.out: TASK [testnode : Create the teuthology user.] ********************************** 2019-12-21T15:41:04.603 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-21T15:41:04.644 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:41:04 +0000 (0:00:00.068) 0:02:42.652 ***** 2019-12-21T15:41:04.670 INFO:teuthology.task.ansible.out: TASK [testnode : Delete the teuthology users password.] ************************ 2019-12-21T15:41:04.670 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-21T15:41:04.715 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:41:04 +0000 (0:00:00.071) 0:02:42.723 ***** 2019-12-21T15:41:04.929 INFO:teuthology.task.ansible.out: TASK [testnode : Add a user for xfstests to test user quotas.] ***************** ok: [smithi172.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 10101, "home": "/home/fsgqa", "move_home": false, "name": "fsgqa", "shell": "/bin/bash", "state": "present", "uid": 10101} 2019-12-21T15:41:04.967 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:41:04 +0000 (0:00:00.251) 0:02:42.975 ***** 2019-12-21T15:41:05.514 INFO:teuthology.task.ansible.out: TASK [testnode : Set a high max open files limit for the teuthology user.] ***** 2019-12-21T15:41:05.515 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => {"changed": false, "checksum": "dc140f4bb26594b8d5e0753bab8d93c0cced7772", "dest": "/etc/security/limits.d/ubuntu.conf", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/security/limits.d/ubuntu.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 112, "state": "file", "uid": 0} 2019-12-21T15:41:05.555 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:41:05 +0000 (0:00:00.587) 0:02:43.562 ***** 2019-12-21T15:41:05.592 INFO:teuthology.task.ansible.out: TASK [testnode : Set hostname var] ********************************************* ok: [smithi172.front.sepia.ceph.com] => {"ansible_facts": {"hostname": "smithi172"}, "changed": false} 2019-12-21T15:41:05.628 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:41:05 +0000 (0:00:00.073) 0:02:43.636 ***** 2019-12-21T15:41:06.478 INFO:teuthology.task.ansible.out: TASK [testnode : Set the system's hostname] ************************************ 2019-12-21T15:41:06.480 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => {"ansible_facts": {"ansible_domain": "", "ansible_fqdn": "smithi172", "ansible_hostname": "smithi172", "ansible_nodename": "smithi172"}, "changed": false, "name": "smithi172"} 2019-12-21T15:41:06.518 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:41:06 +0000 (0:00:00.889) 0:02:44.525 ***** 2019-12-21T15:41:06.993 INFO:teuthology.task.ansible.out: TASK [testnode : Upload distro major version specific sshd_config] ************* 2019-12-21T15:41:06.994 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => {"changed": false, "checksum": "a5e550b2aa74f6355e926d70d73a50ad9ab4b4be", "dest": "/etc/ssh/sshd_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/sshd_config", "secontext": "system_u:object_r:etc_t:s0", "size": 1043, "state": "file", "uid": 0} 2019-12-21T15:41:07.032 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:41:07 +0000 (0:00:00.514) 0:02:45.039 ***** 2019-12-21T15:41:07.507 INFO:teuthology.task.ansible.out: TASK [testnode : Upload ssh_config] ******************************************** 2019-12-21T15:41:07.507 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => {"changed": false, "checksum": "58d78b02605c8368f1913de12e145f4ad930fafc", "dest": "/etc/ssh/ssh_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/ssh_config", "secontext": "system_u:object_r:etc_t:s0", "size": 570, "state": "file", "uid": 0} 2019-12-21T15:41:07.555 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:41:07 +0000 (0:00:00.522) 0:02:45.562 ***** 2019-12-21T15:41:08.184 INFO:teuthology.task.ansible.out: TASK [testnode : Add ssh pubkeys] ********************************************** 2019-12-21T15:41:08.186 INFO:teuthology.task.ansible.out:changed: [smithi172.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "comment": null, "exclusive": false, "follow": false, "key": "https://raw.githubusercontent.com/ceph/keys/autogenerated/ssh/@all.pub", "key_options": null, "keyfile": "/home/ubuntu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ubuntu", "validate_certs": true} 2019-12-21T15:41:08.223 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:41:08 +0000 (0:00:00.668) 0:02:46.230 ***** 2019-12-21T15:41:08.706 INFO:teuthology.task.ansible.out: TASK [testnode : Create remote.conf] ******************************************* 2019-12-21T15:41:08.707 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => {"changed": false, "checksum": "68d0b8ee2f88a77895f257aa69f32a42e47f455b", "dest": "/etc/security/limits.d/remote.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/security/limits.d/remote.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 116, "state": "file", "uid": 0} 2019-12-21T15:41:08.770 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:41:08 +0000 (0:00:00.547) 0:02:46.778 ***** 2019-12-21T15:41:08.976 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2019-12-21T15:41:08.976 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => {"changed": false, "dest": "/etc/fuse.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "secontext": "unconfined_u:object_r:etc_t:s0", "size": 0, "state": "file", "uid": 0} 2019-12-21T15:41:09.014 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:41:09 +0000 (0:00:00.243) 0:02:47.021 ***** 2019-12-21T15:41:09.219 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2019-12-21T15:41:09.221 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => {"changed": false, "gid": 36, "name": "kvm", "state": "present", "system": false} 2019-12-21T15:41:09.257 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:41:09 +0000 (0:00:00.243) 0:02:47.265 ***** 2019-12-21T15:41:09.481 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2019-12-21T15:41:09.483 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => {"append": true, "changed": false, "comment": "", "group": 1205, "groups": "kvm,disk", "home": "/home/ubuntu", "move_home": false, "name": "ubuntu", "shell": "/bin/bash", "state": "present", "uid": 1000} 2019-12-21T15:41:09.519 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:41:09 +0000 (0:00:00.261) 0:02:47.527 ***** 2019-12-21T15:41:10.000 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2019-12-21T15:41:10.002 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => {"changed": false, "checksum": "be3223c8b0917bde02657a5f9ddd95277cc3f812", "dest": "/etc/sudoers", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers", "secontext": "system_u:object_r:etc_t:s0", "size": 1808, "state": "file", "uid": 0} 2019-12-21T15:41:10.042 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:41:10 +0000 (0:00:00.522) 0:02:48.049 ***** 2019-12-21T15:41:10.538 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2019-12-21T15:41:10.540 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => {"changed": false, "checksum": "12a2e17405b9e00ea2061e25929905d08e5fe19f", "dest": "/etc/security/limits.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/security/limits.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 854, "state": "file", "uid": 0} 2019-12-21T15:41:10.579 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:41:10 +0000 (0:00:00.537) 0:02:48.587 ***** 2019-12-21T15:41:12.242 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ [WARNING]: Consider using the yum, dnf or zypper module rather than running 'rpm'. If you need to use command because yum, dnf or zypper is insufficient you can add 'warn: false' to this command task or set 'command_warnings=False' in ansible.cfg to get rid of this message. changed: [smithi172.front.sepia.ceph.com] => {"changed": true, "cmd": ["rpm", "--rebuilddb"], "delta": "0:00:01.450312", "end": "2019-12-21 15:41:12.209465", "rc": 0, "start": "2019-12-21 15:41:10.759153", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2019-12-21T15:41:12.288 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:41:12 +0000 (0:00:01.709) 0:02:50.296 ***** 2019-12-21T15:41:12.855 INFO:teuthology.task.ansible.out: TASK [testnode : Configure common additional repos in /etc/yum.repos.d/] ******* changed: [smithi172.front.sepia.ceph.com] => (item={'value': {u'gpgcheck': 0, u'enabled': 1, u'name': u'lab-extras', u'baseurl': u'http://apt-mirror.front.sepia.ceph.com/lab-extras/8/'}, 'key': u'lab-extras'}) => {"ansible_loop_var": "item", "changed": true, "checksum": "a0e2ac0628a264e14b96b1c57627a3aa344a6c52", "dest": "/etc/yum.repos.d/lab-extras.repo", "gid": 0, "group": "root", "item": {"key": "lab-extras", "value": {"baseurl": "http://apt-mirror.front.sepia.ceph.com/lab-extras/8/", "enabled": 1, "gpgcheck": 0, "name": "lab-extras"}}, "md5sum": "fed359af26a9f925305897bc0401b3aa", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:system_conf_t:s0", "size": 203, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1576942872.32-43409486443622/source", "state": "file", "uid": 0} 2019-12-21T15:41:12.903 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:41:12 +0000 (0:00:00.612) 0:02:50.908 ***** 2019-12-21T15:41:12.927 INFO:teuthology.task.ansible.out: TASK [testnode : Configure version specific repos in /etc/yum.repos.d/] ******** 2019-12-21T15:41:12.970 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:41:12 +0000 (0:00:00.069) 0:02:50.978 ***** 2019-12-21T15:41:14.168 INFO:teuthology.task.ansible.out: TASK [testnode : Enable copr repos] ******************************************** 2019-12-21T15:41:14.169 INFO:teuthology.task.ansible.out:changed: [smithi172.front.sepia.ceph.com] => (item=ktdreyer/ceph-el8) => {"ansible_loop_var": "item", "changed": true, "cmd": ["dnf", "-y", "copr", "enable", "ktdreyer/ceph-el8"], "delta": "0:00:00.972065", "end": "2019-12-21 15:41:14.134708", "item": "ktdreyer/ceph-el8", "rc": 0, "start": "2019-12-21 15:41:13.162643", "stderr": "", "stderr_lines": [], "stdout": "Updating Subscription Management repositories.\nRepository successfully enabled.", "stdout_lines": ["Updating Subscription Management repositories.", "Repository successfully enabled."], "warnings": ["Consider using the dnf module rather than running 'dnf'. If you need to use command because dnf is insufficient you can add 'warn: false' to this command task or set 'command_warnings=False' in ansible.cfg to get rid of this message."]} 2019-12-21T15:41:14.170 INFO:teuthology.task.ansible.out: [WARNING]: Consider using the dnf module rather than running 'dnf'. If you need to use command because dnf is insufficient you can add 'warn: false' to this command task or set 'command_warnings=False' in ansible.cfg to get rid of this message. 2019-12-21T15:41:14.211 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:41:14 +0000 (0:00:01.240) 0:02:52.218 ***** 2019-12-21T15:41:14.247 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG] ************************************************** 2019-12-21T15:41:14.249 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-21T15:41:14.306 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:41:14 +0000 (0:00:00.094) 0:02:52.313 ***** 2019-12-21T15:41:14.342 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG keys] ********************************************* 2019-12-21T15:41:14.344 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} 2019-12-21T15:41:14.354 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} 2019-12-21T15:41:14.396 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:41:14 +0000 (0:00:00.090) 0:02:52.404 ***** 2019-12-21T15:41:15.432 INFO:teuthology.task.ansible.out: TASK [testnode : Clean yum cache] ********************************************** 2019-12-21T15:41:15.432 INFO:teuthology.task.ansible.out: [WARNING]: Consider using the yum module rather than running 'yum'. If you need to use command because yum is insufficient you can add 'warn: false' to this command task or set 'command_warnings=False' in ansible.cfg to get rid of this message. 2019-12-21T15:41:15.434 INFO:teuthology.task.ansible.out:changed: [smithi172.front.sepia.ceph.com] => {"changed": true, "cmd": "yum clean all", "delta": "0:00:00.831890", "end": "2019-12-21 15:41:15.406154", "rc": 0, "start": "2019-12-21 15:41:14.574264", "stderr": "", "stderr_lines": [], "stdout": "Updating Subscription Management repositories.\n31 files removed", "stdout_lines": ["Updating Subscription Management repositories.", "31 files removed"]} 2019-12-21T15:41:15.483 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:41:15 +0000 (0:00:01.086) 0:02:53.490 ***** 2019-12-21T15:41:15.512 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2019-12-21T15:41:15.514 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-21T15:41:15.560 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:41:15 +0000 (0:00:00.076) 0:02:53.567 ***** 2019-12-21T15:41:15.596 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure yum_utils is present.] ********************************* 2019-12-21T15:41:15.597 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-21T15:41:15.635 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:41:15 +0000 (0:00:00.075) 0:02:53.643 ***** 2019-12-21T15:41:15.671 INFO:teuthology.task.ansible.out: TASK [testnode : Removing saved yum transactions] ****************************** 2019-12-21T15:41:15.672 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-21T15:41:15.713 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:41:15 +0000 (0:00:00.077) 0:02:53.720 ***** 2019-12-21T15:41:15.921 INFO:teuthology.task.ansible.out: TASK [testnode : Check if ceph-debuginfo is installed] ************************* 2019-12-21T15:41:15.922 INFO:teuthology.task.ansible.out:fatal: [smithi172.front.sepia.ceph.com]: FAILED! => {"changed": false, "cmd": ["rpm", "-q", "ceph-debuginfo"], "delta": "0:00:00.006481", "end": "2019-12-21 15:41:15.898386", "msg": "non-zero return code", "rc": 1, "start": "2019-12-21 15:41:15.891905", "stderr": "", "stderr_lines": [], "stdout": "package ceph-debuginfo is not installed", "stdout_lines": ["package ceph-debuginfo is not installed"]} ...ignoring 2019-12-21T15:41:15.961 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:41:15 +0000 (0:00:00.248) 0:02:53.969 ***** 2019-12-21T15:41:15.997 INFO:teuthology.task.ansible.out: TASK [testnode : Work around https://bugzilla.redhat.com/show_bug.cgi?id=1234967] *** 2019-12-21T15:41:15.999 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-21T15:41:16.035 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:41:16 +0000 (0:00:00.073) 0:02:54.043 ***** 2019-12-21T15:41:25.973 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2019-12-21T15:41:25.973 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2019-12-21T15:41:26.013 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:41:26 +0000 (0:00:09.977) 0:03:04.020 ***** 2019-12-21T15:41:30.948 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2019-12-21T15:41:30.949 INFO:teuthology.task.ansible.out:changed: [smithi172.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Removed: hdparm-9.54-2.el8.x86_64", "Removed: tuned-2.10.0-15.el8.noarch"]} 2019-12-21T15:41:30.990 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:41:30 +0000 (0:00:04.976) 0:03:08.997 ***** 2019-12-21T15:41:50.922 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2019-12-21T15:41:50.924 INFO:teuthology.task.ansible.out:changed: [smithi172.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Group core installed.", "Group base installed.", "Installed: dnf-utils", "Installed: git-all", "Installed: sysstat", "Installed: libedit", "Installed: boost-thread", "Installed: xfsprogs", "Installed: gdisk", "Installed: parted", "Installed: libgcrypt", "Installed: fuse-libs", "Installed: openssl", "Installed: libuuid", "Installed: attr", "Installed: ant", "Installed: lsof", "Installed: gettext", "Installed: bc", "Installed: xfsdump", "Installed: blktrace", "Installed: usbredir", "Installed: podman", "Installed: libev-devel", "Installed: ncurses-devel", "Installed: gcc", "Installed: git", "Installed: python3-nose", "Installed: python3-virtualenv", "Installed: genisoimage", "Installed: qemu-img", "Installed: libacl-devel", "Installed: dbench", "Installed: autoconf", "Installed: glibc-headers-2.28-42.el8_0.1.x86_64", "Installed: platform-python-devel-3.6.8-1.el8.x86_64", "Installed: glibc-devel-2.28-42.el8_0.1.x86_64", "Installed: python3-virtualenv-15.1.0-18.module+el8.0.0+2966+d39a1df3.noarch", "Installed: libattr-devel-2.4.48-3.el8.x86_64", "Installed: python-rpm-macros-3-37.el8.noarch", "Installed: hdparm-9.54-2.el8.x86_64", "Installed: annobin-8.64-1.el8.x86_64", "Installed: tuned-2.10.0-15.el8.noarch", "Installed: kernel-headers-4.18.0-80.11.2.el8_0.x86_64", "Installed: libxcrypt-devel-4.1.1-4.el8.x86_64", "Installed: libacl-devel-2.2.53-1.el8.x86_64", "Installed: qemu-img-15:2.12.0-65.module+el8.0.0+4084+cceb9f44.5.x86_64", "Installed: libusal-1.1.11-39.el8.x86_64", "Installed: python3-nose-1.3.7-30.module+el8.0.0+2966+d39a1df3.noarch", "Installed: cpp-8.2.1-3.5.el8.x86_64", "Installed: isl-0.16.1-6.el8.x86_64", "Installed: autoconf-2.69-27.el8.noarch", "Installed: python3-rpm-generators-5-4.el8.noarch", "Installed: dbench-4.0-20.el8.x86_64", "Installed: gcc-8.2.1-3.5.el8.x86_64", "Installed: python36-devel-3.6.8-2.module+el8.0.0+2975+e0f02136.x86_64", "Installed: li 2019-12-21T15:41:50.925 INFO:teuthology.task.ansible.out:bmpc-1.0.2-9.el8.x86_64", "Installed: genisoimage-1.1.11-39.el8.x86_64"]} 2019-12-21T15:41:50.968 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:41:50 +0000 (0:00:19.974) 0:03:28.972 ***** 2019-12-21T15:41:50.997 INFO:teuthology.task.ansible.out: TASK [testnode : Install epel packages] **************************************** 2019-12-21T15:41:50.999 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-21T15:41:51.054 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:41:51 +0000 (0:00:00.089) 0:03:29.062 ***** 2019-12-21T15:41:52.675 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2019-12-21T15:41:52.677 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2019-12-21T15:41:52.712 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:41:52 +0000 (0:00:01.658) 0:03:30.720 ***** 2019-12-21T15:41:52.739 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2019-12-21T15:41:52.741 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-21T15:41:52.777 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:41:52 +0000 (0:00:00.064) 0:03:30.784 ***** 2019-12-21T15:41:53.091 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable firewalld] *********************************** 2019-12-21T15:41:53.093 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => {"changed": false, "enabled": false, "name": "firewalld", "state": "stopped", "status": {"ActiveEnterTimestampMonotonic": "0", "ActiveExitTimestampMonotonic": "0", "ActiveState": "inactive", "After": "basic.target sysinit.target dbus.service dbus.socket system.slice polkit.service", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "no", "AssertTimestampMonotonic": "0", "Before": "shutdown.target network-pre.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "BusName": "org.fedoraproject.FirewallD1", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "no", "ConditionTimestampMonotonic": "0", "ConfigurationDirectoryMode": "0755", "Conflicts": "ipset.service shutdown.target ebtables.service ip6tables.service iptables.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "firewalld - dynamic firewall daemon", "DevicePolicy": "auto", "Documentation": "man:firewalld(1)", "DynamicUser": "no", "EnvironmentFiles": "/etc/sysconfig/firewalld (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTim 2019-12-21T15:41:53.094 INFO:teuthology.task.ansible.out:estampMonotonic": "0", "ExecMainPID": "0", "ExecMainStartTimestampMonotonic": "0", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/firewalld ; argv[]=/usr/sbin/firewalld --nofork --nopid $FIREWALLD_ARGS ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/firewalld.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "firewalld.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestampMonotonic": "0", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "mixed", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "infinity", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "16777216", "LimitMEMLOCKSoft": "16777216", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "127463", "LimitNPROCSoft": "127463", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127463", 2019-12-21T15:41:53.096 INFO:teuthology.task.ansible.out:"LimitSIGPENDINGSoft": "127463", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "0", "MemoryAccounting": "yes", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "firewalld.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateMounts": "no", "PrivateNetwork": "no", "PrivateTmp": "no", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "system.slice sysinit.target dbus.socket", "Restart": "no", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "null", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "null", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestampMonotonic": "0", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "dead", "SuccessAction": "none", "SyslogFacility": "3 2019-12-21T15:41:53.097 INFO:teuthology.task.ansible.out:", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "[not set]", "TasksMax": "26213", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "dbus", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "disabled", "UtmpMode": "init", "Wants": "network-pre.target", "WatchdogTimestampMonotonic": "0", "WatchdogUSec": "0"}} 2019-12-21T15:41:53.131 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:41:53 +0000 (0:00:00.353) 0:03:31.138 ***** 2019-12-21T15:41:53.432 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable iptables] ************************************ 2019-12-21T15:41:53.433 INFO:teuthology.task.ansible.out:fatal: [smithi172.front.sepia.ceph.com]: FAILED! => {"changed": false, "msg": "Could not find the requested service iptables: host"} ...ignoring 2019-12-21T15:41:53.472 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:41:53 +0000 (0:00:00.340) 0:03:31.479 ***** 2019-12-21T15:41:54.169 INFO:teuthology.task.ansible.out: TASK [testnode : Enable SELinux] *********************************************** ok: [smithi172.front.sepia.ceph.com] => {"changed": false, "configfile": "/etc/selinux/config", "msg": "", "policy": "targeted", "reboot_required": false, "state": "permissive"} 2019-12-21T15:41:54.211 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:41:54 +0000 (0:00:00.739) 0:03:32.219 ***** 2019-12-21T15:41:54.239 INFO:teuthology.task.ansible.out: TASK [testnode : Check for custom repos] *************************************** 2019-12-21T15:41:54.241 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-21T15:41:54.283 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:41:54 +0000 (0:00:00.071) 0:03:32.291 ***** 2019-12-21T15:41:54.309 INFO:teuthology.task.ansible.out: TASK [testnode : Remove custom repos] ****************************************** 2019-12-21T15:41:54.351 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:41:54 +0000 (0:00:00.068) 0:03:32.359 ***** 2019-12-21T15:41:54.377 INFO:teuthology.task.ansible.out: TASK [testnode : Set apt preferences] ****************************************** 2019-12-21T15:41:54.379 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-21T15:41:54.417 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:41:54 +0000 (0:00:00.065) 0:03:32.424 ***** 2019-12-21T15:41:54.446 INFO:teuthology.task.ansible.out: TASK [testnode : Add sources list] ********************************************* 2019-12-21T15:41:54.447 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-21T15:41:54.491 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:41:54 +0000 (0:00:00.074) 0:03:32.499 ***** 2019-12-21T15:41:54.492 INFO:teuthology.task.ansible.out: 2019-12-21T15:41:54.517 INFO:teuthology.task.ansible.out: TASK [testnode : Install apt keys] ********************************************* 2019-12-21T15:41:54.518 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} 2019-12-21T15:41:54.529 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} 2019-12-21T15:41:54.571 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:41:54 +0000 (0:00:00.079) 0:03:32.578 ***** 2019-12-21T15:41:54.596 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-apt] ******************************************* 2019-12-21T15:41:54.598 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-21T15:41:54.635 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:41:54 +0000 (0:00:00.064) 0:03:32.643 ***** 2019-12-21T15:41:54.661 INFO:teuthology.task.ansible.out: TASK [testnode : Add local apt repos.] ***************************************** 2019-12-21T15:41:54.702 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:41:54 +0000 (0:00:00.066) 0:03:32.709 ***** 2019-12-21T15:41:54.732 INFO:teuthology.task.ansible.out: TASK [testnode : Blast the apt cache.] ***************************************** 2019-12-21T15:41:54.734 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-21T15:41:54.771 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:41:54 +0000 (0:00:00.069) 0:03:32.778 ***** 2019-12-21T15:41:54.807 INFO:teuthology.task.ansible.out: TASK [testnode : Update apt cache.] ******************************************** 2019-12-21T15:41:54.808 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-21T15:41:54.852 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:41:54 +0000 (0:00:00.076) 0:03:32.855 ***** 2019-12-21T15:41:54.886 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure packages are not present.] ***************************** 2019-12-21T15:41:54.888 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-21T15:41:54.925 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:41:54 +0000 (0:00:00.077) 0:03:32.933 ***** 2019-12-21T15:41:54.951 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2019-12-21T15:41:54.953 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-21T15:41:54.995 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:41:54 +0000 (0:00:00.069) 0:03:33.003 ***** 2019-12-21T15:41:55.020 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade non aarch64 packages] ********************************* 2019-12-21T15:41:55.022 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-21T15:41:55.060 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:41:55 +0000 (0:00:00.064) 0:03:33.067 ***** 2019-12-21T15:41:55.087 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2019-12-21T15:41:55.089 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-21T15:41:55.126 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:41:55 +0000 (0:00:00.066) 0:03:33.134 ***** 2019-12-21T15:41:55.152 INFO:teuthology.task.ansible.out: TASK [testnode : Install non aarch64 packages] ********************************* 2019-12-21T15:41:55.153 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-21T15:41:55.197 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:41:55 +0000 (0:00:00.070) 0:03:33.204 ***** 2019-12-21T15:41:55.223 INFO:teuthology.task.ansible.out: TASK [testnode : Use perl to add user_xattr to the root mount options in fstab.] *** 2019-12-21T15:41:55.224 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-21T15:41:55.262 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:41:55 +0000 (0:00:00.065) 0:03:33.270 ***** 2019-12-21T15:41:55.288 INFO:teuthology.task.ansible.out: TASK [testnode : Enable xattr for this boot.] ********************************** 2019-12-21T15:41:55.289 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-21T15:41:55.327 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:41:55 +0000 (0:00:00.064) 0:03:33.334 ***** 2019-12-21T15:41:55.353 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure fuse group exists.] ************************************ 2019-12-21T15:41:55.355 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-21T15:41:55.397 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:41:55 +0000 (0:00:00.069) 0:03:33.404 ***** 2019-12-21T15:41:55.422 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/fuse.conf.] *************************************** 2019-12-21T15:41:55.424 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-21T15:41:55.464 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:41:55 +0000 (0:00:00.066) 0:03:33.471 ***** 2019-12-21T15:41:55.500 INFO:teuthology.task.ansible.out: TASK [testnode : Add teuthology user to groups fuse, kvm and disk.] ************ 2019-12-21T15:41:55.502 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-21T15:41:55.538 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:41:55 +0000 (0:00:00.074) 0:03:33.546 ***** 2019-12-21T15:41:55.564 INFO:teuthology.task.ansible.out: TASK [testnode : Set up static IP in /etc/hosts.] ****************************** 2019-12-21T15:41:55.566 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-21T15:41:55.611 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:41:55 +0000 (0:00:00.072) 0:03:33.618 ***** 2019-12-21T15:41:55.637 INFO:teuthology.task.ansible.out: TASK [testnode : Stop apache2] ************************************************* 2019-12-21T15:41:55.639 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-21T15:41:55.679 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:41:55 +0000 (0:00:00.068) 0:03:33.687 ***** 2019-12-21T15:41:55.706 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2019-12-21T15:41:55.707 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-21T15:41:55.750 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:41:55 +0000 (0:00:00.070) 0:03:33.757 ***** 2019-12-21T15:41:55.775 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2019-12-21T15:41:55.777 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-21T15:41:55.814 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:41:55 +0000 (0:00:00.063) 0:03:33.821 ***** 2019-12-21T15:41:55.841 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-21T15:41:55.903 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:41:55 +0000 (0:00:00.089) 0:03:33.911 ***** 2019-12-21T15:41:55.929 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2019-12-21T15:41:55.930 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-21T15:41:55.971 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:41:55 +0000 (0:00:00.067) 0:03:33.978 ***** 2019-12-21T15:41:55.997 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2019-12-21T15:41:55.999 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-21T15:41:56.035 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:41:56 +0000 (0:00:00.064) 0:03:34.043 ***** 2019-12-21T15:41:56.061 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2019-12-21T15:41:56.062 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-21T15:41:56.101 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:41:56 +0000 (0:00:00.065) 0:03:34.108 ***** 2019-12-21T15:41:56.126 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2019-12-21T15:41:56.127 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-21T15:41:56.167 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:41:56 +0000 (0:00:00.066) 0:03:34.175 ***** 2019-12-21T15:41:56.213 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2019-12-21T15:41:56.215 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-21T15:41:56.251 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:41:56 +0000 (0:00:00.084) 0:03:34.259 ***** 2019-12-21T15:41:56.279 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-21T15:41:56.315 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:41:56 +0000 (0:00:00.063) 0:03:34.323 ***** 2019-12-21T15:41:56.341 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2019-12-21T15:41:56.342 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-21T15:41:56.381 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:41:56 +0000 (0:00:00.066) 0:03:34.389 ***** 2019-12-21T15:41:56.417 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2019-12-21T15:41:56.419 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-21T15:41:56.460 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:41:56 +0000 (0:00:00.078) 0:03:34.468 ***** 2019-12-21T15:41:56.486 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2019-12-21T15:41:56.487 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-21T15:41:56.527 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:41:56 +0000 (0:00:00.066) 0:03:34.534 ***** 2019-12-21T15:41:56.553 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2019-12-21T15:41:56.554 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-21T15:41:56.591 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:41:56 +0000 (0:00:00.063) 0:03:34.598 ***** 2019-12-21T15:41:56.617 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2019-12-21T15:41:56.618 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-21T15:41:56.656 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:41:56 +0000 (0:00:00.065) 0:03:34.663 ***** 2019-12-21T15:41:56.681 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2019-12-21T15:41:56.683 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-21T15:41:56.720 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:41:56 +0000 (0:00:00.063) 0:03:34.727 ***** 2019-12-21T15:41:56.749 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2019-12-21T15:41:56.750 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-21T15:41:56.786 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:41:56 +0000 (0:00:00.066) 0:03:34.794 ***** 2019-12-21T15:41:56.824 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2019-12-21T15:41:56.825 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-21T15:41:56.882 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:41:56 +0000 (0:00:00.096) 0:03:34.890 ***** 2019-12-21T15:41:56.918 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2019-12-21T15:41:56.920 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-21T15:41:56.959 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:41:56 +0000 (0:00:00.076) 0:03:34.966 ***** 2019-12-21T15:41:56.986 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2019-12-21T15:41:56.987 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-21T15:41:57.025 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:41:57 +0000 (0:00:00.066) 0:03:35.032 ***** 2019-12-21T15:41:57.050 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2019-12-21T15:41:57.052 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-21T15:41:57.089 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:41:57 +0000 (0:00:00.062) 0:03:35.095 ***** 2019-12-21T15:41:57.116 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-21T15:41:57.152 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:41:57 +0000 (0:00:00.064) 0:03:35.160 ***** 2019-12-21T15:41:57.178 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2019-12-21T15:41:57.179 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-21T15:41:57.218 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:41:57 +0000 (0:00:00.065) 0:03:35.225 ***** 2019-12-21T15:41:57.243 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2019-12-21T15:41:57.245 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-21T15:41:57.281 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:41:57 +0000 (0:00:00.063) 0:03:35.289 ***** 2019-12-21T15:41:57.307 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2019-12-21T15:41:57.308 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-21T15:41:57.347 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:41:57 +0000 (0:00:00.065) 0:03:35.354 ***** 2019-12-21T15:41:57.372 INFO:teuthology.task.ansible.out: TASK [testnode : Set grub config.] ********************************************* 2019-12-21T15:41:57.374 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-21T15:41:57.415 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:41:57 +0000 (0:00:00.068) 0:03:35.423 ***** 2019-12-21T15:41:57.442 INFO:teuthology.task.ansible.out: TASK [testnode : Remove /etc/ceph] ********************************************* 2019-12-21T15:41:57.443 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-21T15:41:57.480 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:41:57 +0000 (0:00:00.065) 0:03:35.488 ***** 2019-12-21T15:41:57.506 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2019-12-21T15:41:57.508 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-21T15:41:57.547 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:41:57 +0000 (0:00:00.066) 0:03:35.555 ***** 2019-12-21T15:41:57.573 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2019-12-21T15:41:57.575 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-21T15:41:57.611 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:41:57 +0000 (0:00:00.063) 0:03:35.618 ***** 2019-12-21T15:41:57.637 INFO:teuthology.task.ansible.out: TASK [testnode : Upload weekly kernel-clean crontab.] ************************** 2019-12-21T15:41:57.638 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-21T15:41:57.675 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:41:57 +0000 (0:00:00.063) 0:03:35.682 ***** 2019-12-21T15:41:57.702 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/grub.d/02_force_timeout.] ************************* 2019-12-21T15:41:57.704 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-21T15:41:57.740 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:41:57 +0000 (0:00:00.065) 0:03:35.748 ***** 2019-12-21T15:41:57.766 INFO:teuthology.task.ansible.out: TASK [testnode : Enable kernel modules to load at boot time.] ****************** skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-21T15:41:57.806 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:41:57 +0000 (0:00:00.065) 0:03:35.813 ***** 2019-12-21T15:41:57.832 INFO:teuthology.task.ansible.out: TASK [testnode : Enabling auto-fsck fix to prevent boot hangup.] *************** 2019-12-21T15:41:57.833 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-21T15:41:57.869 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:41:57 +0000 (0:00:00.063) 0:03:35.877 ***** 2019-12-21T15:41:57.896 INFO:teuthology.task.ansible.out: TASK [testnode : Work around broken wget on wheezy.] *************************** 2019-12-21T15:41:57.898 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-21T15:41:57.934 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:41:57 +0000 (0:00:00.064) 0:03:35.941 ***** 2019-12-21T15:41:57.960 INFO:teuthology.task.ansible.out: TASK [testnode : Stop collectl] ************************************************ 2019-12-21T15:41:57.961 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-21T15:41:58.001 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:41:57 +0000 (0:00:00.067) 0:03:36.008 ***** 2019-12-21T15:41:58.037 INFO:teuthology.task.ansible.out: TASK [testnode : Add PATH to the teuthology_user .bashrc.] ********************* 2019-12-21T15:41:58.039 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-21T15:41:58.080 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:41:58 +0000 (0:00:00.079) 0:03:36.088 ***** 2019-12-21T15:41:58.107 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if we need to edit /etc/profile.] **************** 2019-12-21T15:41:58.108 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-21T15:41:58.147 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:41:58 +0000 (0:00:00.066) 0:03:36.155 ***** 2019-12-21T15:41:58.173 INFO:teuthology.task.ansible.out: TASK [testnode : Update /etc/profile if needed.] ******************************* 2019-12-21T15:41:58.175 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-21T15:41:58.211 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:41:58 +0000 (0:00:00.063) 0:03:36.218 ***** 2019-12-21T15:41:58.236 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2019-12-21T15:41:58.238 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => {"ansible_facts": {"nvme_card_present": false}, "changed": false} 2019-12-21T15:41:58.273 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:41:58 +0000 (0:00:00.062) 0:03:36.281 ***** 2019-12-21T15:41:58.320 INFO:teuthology.task.ansible.out: TASK [testnode : Check for NVMe drive] ***************************************** 2019-12-21T15:41:58.321 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item=sda) => {"ansible_loop_var": "item", "changed": false, "item": "sda", "skip_reason": "Conditional result was False"} 2019-12-21T15:41:58.331 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item=nvme0n1) => {"ansible_facts": {"nvme_card_present": true}, "ansible_loop_var": "item", "changed": false, "item": "nvme0n1"} 2019-12-21T15:41:58.353 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item=dm-4) => {"ansible_loop_var": "item", "changed": false, "item": "dm-4", "skip_reason": "Conditional result was False"} 2019-12-21T15:41:58.355 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item=dm-2) => {"ansible_loop_var": "item", "changed": false, "item": "dm-2", "skip_reason": "Conditional result was False"} skipping: [smithi172.front.sepia.ceph.com] => (item=dm-3) => {"ansible_loop_var": "item", "changed": false, "item": "dm-3", "skip_reason": "Conditional result was False"} 2019-12-21T15:41:58.365 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item=dm-0) => {"ansible_loop_var": "item", "changed": false, "item": "dm-0", "skip_reason": "Conditional result was False"} 2019-12-21T15:41:58.366 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item=dm-1) => {"ansible_loop_var": "item", "changed": false, "item": "dm-1", "skip_reason": "Conditional result was False"} 2019-12-21T15:41:58.415 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:41:58 +0000 (0:00:00.141) 0:03:36.422 ***** 2019-12-21T15:41:58.561 INFO:teuthology.task.ansible.out: TASK [testnode : Check for teuthology-lock command] **************************** 2019-12-21T15:41:58.563 INFO:teuthology.task.ansible.out:changed: [smithi172.front.sepia.ceph.com -> localhost] => {"changed": true, "cmd": "which teuthology-lock", "delta": "0:00:00.004250", "end": "2019-12-21 15:41:58.540930", "rc": 0, "start": "2019-12-21 15:41:58.536680", "stderr": "", "stderr_lines": [], "stdout": "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/bin/teuthology-lock", "stdout_lines": ["/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/bin/teuthology-lock"]} 2019-12-21T15:41:58.602 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:41:58 +0000 (0:00:00.187) 0:03:36.610 ***** 2019-12-21T15:41:58.629 INFO:teuthology.task.ansible.out: TASK [testnode : Mark system down if NVMe card missing] ************************ 2019-12-21T15:41:58.630 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-21T15:41:58.673 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:41:58 +0000 (0:00:00.070) 0:03:36.680 ***** 2019-12-21T15:41:58.710 INFO:teuthology.task.ansible.out: TASK [testnode : Update description in paddles lock DB if NVMe card missing] *** 2019-12-21T15:41:58.711 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-21T15:41:58.749 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:41:58 +0000 (0:00:00.076) 0:03:36.756 ***** 2019-12-21T15:41:58.775 INFO:teuthology.task.ansible.out: TASK [testnode : Fail rest of playbook due to missing NVMe card] *************** 2019-12-21T15:41:58.776 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-21T15:41:58.814 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:41:58 +0000 (0:00:00.064) 0:03:36.821 ***** 2019-12-21T15:41:58.841 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure apt dependencies are installed] ********************* 2019-12-21T15:41:58.841 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-21T15:41:58.881 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:41:58 +0000 (0:00:00.067) 0:03:36.889 ***** 2019-12-21T15:42:00.617 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure rpm dependencies are installed] ********************* ok: [smithi172.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": ["Installed: gdisk", "Installed: device-mapper"]} 2019-12-21T15:42:00.663 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:42:00 +0000 (0:00:01.781) 0:03:38.670 ***** 2019-12-21T15:42:00.700 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2019-12-21T15:42:00.701 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={u'block_used': 12836194, u'uuid': u'cf568be9-1a39-43d2-a003-fbe6239693bc', u'size_total': 983429562368, u'block_total': 240095108, u'mount': u'/', u'block_available': 227258914, u'size_available': 930852511744, u'fstype': u'ext4', u'inode_total': 61054976, u'inode_available': 60993208, u'device': u'/dev/sda1', u'inode_used': 61768, u'block_size': 4096, u'options': u'rw,seclabel,relatime'}) => {"ansible_facts": {"root_disk": "/dev/sda"}, "ansible_loop_var": "item", "changed": false, "item": {"block_available": 227258914, "block_size": 4096, "block_total": 240095108, "block_used": 12836194, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60993208, "inode_total": 61054976, "inode_used": 61768, "mount": "/", "options": "rw,seclabel,relatime", "size_available": 930852511744, "size_total": 983429562368, "uuid": "cf568be9-1a39-43d2-a003-fbe6239693bc"}} 2019-12-21T15:42:00.701 INFO:teuthology.task.ansible.out: 2019-12-21T15:42:00.741 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:42:00 +0000 (0:00:00.077) 0:03:38.748 ***** 2019-12-21T15:42:00.943 INFO:teuthology.task.ansible.out: TASK [testnode : Compile list of non-root partitions] ************************** 2019-12-21T15:42:00.944 INFO:teuthology.task.ansible.out:changed: [smithi172.front.sepia.ceph.com] => {"changed": true, "cmd": "lsblk --list --noheadings | grep part | grep -v sda | awk '{ print $1 }'", "delta": "0:00:00.005221", "end": "2019-12-21 15:42:00.918947", "rc": 0, "start": "2019-12-21 15:42:00.913726", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2019-12-21T15:42:00.982 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:42:00 +0000 (0:00:00.241) 0:03:38.990 ***** 2019-12-21T15:42:01.022 INFO:teuthology.task.ansible.out: TASK [testnode : Unmount any non-root mountpoints] ***************************** 2019-12-21T15:42:01.024 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={u'block_used': 12836194, u'uuid': u'cf568be9-1a39-43d2-a003-fbe6239693bc', u'size_total': 983429562368, u'block_total': 240095108, u'mount': u'/', u'block_available': 227258914, u'size_available': 930852511744, u'fstype': u'ext4', u'inode_total': 61054976, u'inode_available': 60993208, u'device': u'/dev/sda1', u'inode_used': 61768, u'block_size': 4096, u'options': u'rw,seclabel,relatime'}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 227258914, "block_size": 4096, "block_total": 240095108, "block_used": 12836194, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60993208, "inode_total": 61054976, "inode_used": 61768, "mount": "/", "options": "rw,seclabel,relatime", "size_available": 930852511744, "size_total": 983429562368, "uuid": "cf568be9-1a39-43d2-a003-fbe6239693bc"}, "skip_reason": "Conditional result was False"} 2019-12-21T15:42:01.066 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:42:01 +0000 (0:00:00.084) 0:03:39.074 ***** 2019-12-21T15:42:01.092 INFO:teuthology.task.ansible.out: TASK [testnode : Wipe filesystems on non-root partitions] ********************** 2019-12-21T15:42:01.132 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:42:01 +0000 (0:00:00.065) 0:03:39.139 ***** 2019-12-21T15:42:01.171 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2019-12-21T15:42:01.172 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': u'ATA', u'links': {u'masters': [], u'labels': [], u'ids': [u'ata-ST1000NM0033-9ZM173_Z1W5XNKA', u'wwn-0x5000c5009294065e'], u'uuids': []}, u'sas_device_handle': None, u'host': u'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', u'support_discard': u'0', u'serial': u'Z1W5XNKA', u'wwn': u'0x5000c5009294065e', u'holders': [], u'size': u'931.51 GB', u'scheduler_mode': u'mq-deadline', u'rotational': u'1', u'sectors': u'1953525168', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'model': u'ST1000NM0033-9ZM', u'partitions': {u'sda1': {u'start': u'2048', u'sectorsize': 512, u'uuid': u'cf568be9-1a39-43d2-a003-fbe6239693bc', u'sectors': u'1953522688', u'holders': [], u'links': {u'masters': [], u'labels': [], u'ids': [u'ata-ST1000NM0033-9ZM173_Z1W5XNKA-part1', u'wwn-0x5000c5009294065e-part1'], u'uuids': [u'cf568be9-1a39-43d2-a003-fbe6239693bc']}, u'size': u'931.51 GB'}}}, 'key': u'sda'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XNKA", "wwn-0x5000c5009294065e"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XNKA-part1", "wwn-0x5000c5009294065e-part1"], "labels": [], "masters": [], "uuids": ["cf568be9-1a39-43d2-a003-fbe6239693bc"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "cf568be9-1a39-43d2-a003-fbe6239693bc"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5XNKA", "size": "931.51 GB", "support_discard": "0", "vend 2019-12-21T15:42:01.173 INFO:teuthology.task.ansible.out:or": "ATA", "virtual": 1, "wwn": "0x5000c5009294065e"}}, "skip_reason": "Conditional result was False"} 2019-12-21T15:42:03.803 INFO:teuthology.task.ansible.out:changed: [smithi172.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [u'dm-0', u'dm-1', u'dm-2', u'dm-3', u'dm-4'], u'labels': [], u'ids': [u'lvm-pv-uuid-M3E1yd-4qLJ-BJqp-tClY-HPKa-i540-fctM4E', u'nvme-INTEL_SSDPEDMD400G4_CVFT6233005M400BGN', u'nvme-nvme.8086-43564654363233333030354d34303042474e-494e54454c205353445045444d443430304734-00000001'], u'uuids': []}, u'sas_device_handle': None, u'host': u'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', u'support_discard': u'512', u'model': u'INTEL SSDPEDMD400G4', u'size': u'372.61 GB', u'scheduler_mode': u'none', u'rotational': u'0', u'sectors': u'781422768', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [u'vg_nvme-lv_4', u'vg_nvme-lv_1', u'vg_nvme-lv_3', u'vg_nvme-lv_5', u'vg_nvme-lv_2'], u'partitions': {}}, 'key': u'nvme0n1'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/nvme0n1 || sgdisk --zap-all /dev/nvme0n1", "delta": "0:00:02.444779", "end": "2019-12-21 15:42:03.775486", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_4", "vg_nvme-lv_1", "vg_nvme-lv_3", "vg_nvme-lv_5", "vg_nvme-lv_2"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-M3E1yd-4qLJ-BJqp-tClY-HPKa-i540-fctM4E", "nvme-INTEL_SSDPEDMD400G4_CVFT6233005M400BGN", "nvme-nvme.8086-43564654363233333030354d34303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2019-12-21 15:42:01.330707", "stderr": "", "stderr_lines": [], "stdout": "Creating new GP 2019-12-21T15:42:03.805 INFO:teuthology.task.ansible.out:T entries.\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2019-12-21T15:42:05.047 INFO:teuthology.task.ansible.out:changed: [smithi172.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_1', u'dm-uuid-LVM-y1halPOkiUJG3Hdv0eQmk8A7QQ3CY3FDESIRk22UYsDUIFVqAa2SokNTWdtX5mFq'], u'uuids': [u'03dcd4a9-d55d-4b83-b445-78f7dcda1ae0']}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-4'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-4 || sgdisk --zap-all /dev/dm-4", "delta": "0:00:01.043742", "end": "2019-12-21 15:42:05.019260", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-y1halPOkiUJG3Hdv0eQmk8A7QQ3CY3FDESIRk22UYsDUIFVqAa2SokNTWdtX5mFq"], "labels": [], "masters": [], "uuids": ["03dcd4a9-d55d-4b83-b445-78f7dcda1ae0"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2019-12-21 15:42:03.975518", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2019-12-21T15:42:06.308 INFO:teuthology.task.ansible.out:changed: [smithi172.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_3', u'dm-uuid-LVM-y1halPOkiUJG3Hdv0eQmk8A7QQ3CY3FDOE8dcEVtfkYOgIB4g3fY71GKn6erNGsx'], u'uuids': [u'06899635-f942-44a1-b2b1-50517f6aaa3b']}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-2'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-2 || sgdisk --zap-all /dev/dm-2", "delta": "0:00:01.059152", "end": "2019-12-21 15:42:06.279191", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-y1halPOkiUJG3Hdv0eQmk8A7QQ3CY3FDOE8dcEVtfkYOgIB4g3fY71GKn6erNGsx"], "labels": [], "masters": [], "uuids": ["06899635-f942-44a1-b2b1-50517f6aaa3b"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2019-12-21 15:42:05.220039", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2019-12-21T15:42:07.558 INFO:teuthology.task.ansible.out:changed: [smithi172.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_2', u'dm-uuid-LVM-y1halPOkiUJG3Hdv0eQmk8A7QQ3CY3FDbU33qKUhSCW82PUb6Js6s5Pby0QFWUtP'], u'uuids': [u'a008c94a-a8eb-4e67-8453-e60d99b1ef45']}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-3'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-3 || sgdisk --zap-all /dev/dm-3", "delta": "0:00:01.040166", "end": "2019-12-21 15:42:07.521551", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-y1halPOkiUJG3Hdv0eQmk8A7QQ3CY3FDbU33qKUhSCW82PUb6Js6s5Pby0QFWUtP"], "labels": [], "masters": [], "uuids": ["a008c94a-a8eb-4e67-8453-e60d99b1ef45"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2019-12-21 15:42:06.481385", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2019-12-21T15:42:08.817 INFO:teuthology.task.ansible.out:changed: [smithi172.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_5', u'dm-uuid-LVM-y1halPOkiUJG3Hdv0eQmk8A7QQ3CY3FDd1dgZEESwTcW0KdykZgyoZBQvapHhOa0'], u'uuids': [u'e00a3f60-423b-4ccd-a694-e919fa9ef997']}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'14.90 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'31252480', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-0'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-0 || sgdisk --zap-all /dev/dm-0", "delta": "0:00:01.047527", "end": "2019-12-21 15:42:08.771097", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-y1halPOkiUJG3Hdv0eQmk8A7QQ3CY3FDd1dgZEESwTcW0KdykZgyoZBQvapHhOa0"], "labels": [], "masters": [], "uuids": ["e00a3f60-423b-4ccd-a694-e919fa9ef997"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2019-12-21 15:42:07.723570", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2019-12-21T15:42:10.072 INFO:teuthology.task.ansible.out:changed: [smithi172.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_4', u'dm-uuid-LVM-y1halPOkiUJG3Hdv0eQmk8A7QQ3CY3FDNmLCTkMonvtl9l3u6cs0sUW0DT7hTrRT'], u'uuids': [u'a04fd9b3-1412-4b03-b7c5-74c7eb4b57d2']}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-1'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-1 || sgdisk --zap-all /dev/dm-1", "delta": "0:00:01.044136", "end": "2019-12-21 15:42:10.030470", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-y1halPOkiUJG3Hdv0eQmk8A7QQ3CY3FDNmLCTkMonvtl9l3u6cs0sUW0DT7hTrRT"], "labels": [], "masters": [], "uuids": ["a04fd9b3-1412-4b03-b7c5-74c7eb4b57d2"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2019-12-21 15:42:08.986334", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2019-12-21T15:42:10.110 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:42:10 +0000 (0:00:08.978) 0:03:48.118 ***** 2019-12-21T15:42:10.147 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2019-12-21T15:42:10.148 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': u'ATA', u'links': {u'masters': [], u'labels': [], u'ids': [u'ata-ST1000NM0033-9ZM173_Z1W5XNKA', u'wwn-0x5000c5009294065e'], u'uuids': []}, u'sas_device_handle': None, u'host': u'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', u'support_discard': u'0', u'serial': u'Z1W5XNKA', u'wwn': u'0x5000c5009294065e', u'holders': [], u'size': u'931.51 GB', u'scheduler_mode': u'mq-deadline', u'rotational': u'1', u'sectors': u'1953525168', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'model': u'ST1000NM0033-9ZM', u'partitions': {u'sda1': {u'start': u'2048', u'sectorsize': 512, u'uuid': u'cf568be9-1a39-43d2-a003-fbe6239693bc', u'sectors': u'1953522688', u'holders': [], u'links': {u'masters': [], u'labels': [], u'ids': [u'ata-ST1000NM0033-9ZM173_Z1W5XNKA-part1', u'wwn-0x5000c5009294065e-part1'], u'uuids': [u'cf568be9-1a39-43d2-a003-fbe6239693bc']}, u'size': u'931.51 GB'}}}, 'key': u'sda'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XNKA", "wwn-0x5000c5009294065e"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XNKA-part1", "wwn-0x5000c5009294065e-part1"], "labels": [], "masters": [], "uuids": ["cf568be9-1a39-43d2-a003-fbe6239693bc"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "cf568be9-1a39-43d2-a003-fbe6239693bc"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5XNKA", "size": "931.51 GB", "support_discard": "0", "vend 2019-12-21T15:42:10.151 INFO:teuthology.task.ansible.out:or": "ATA", "virtual": 1, "wwn": "0x5000c5009294065e"}}, "skip_reason": "Conditional result was False"} 2019-12-21T15:42:10.387 INFO:teuthology.task.ansible.out:changed: [smithi172.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [u'dm-0', u'dm-1', u'dm-2', u'dm-3', u'dm-4'], u'labels': [], u'ids': [u'lvm-pv-uuid-M3E1yd-4qLJ-BJqp-tClY-HPKa-i540-fctM4E', u'nvme-INTEL_SSDPEDMD400G4_CVFT6233005M400BGN', u'nvme-nvme.8086-43564654363233333030354d34303042474e-494e54454c205353445045444d443430304734-00000001'], u'uuids': []}, u'sas_device_handle': None, u'host': u'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', u'support_discard': u'512', u'model': u'INTEL SSDPEDMD400G4', u'size': u'372.61 GB', u'scheduler_mode': u'none', u'rotational': u'0', u'sectors': u'781422768', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [u'vg_nvme-lv_4', u'vg_nvme-lv_1', u'vg_nvme-lv_3', u'vg_nvme-lv_5', u'vg_nvme-lv_2'], u'partitions': {}}, 'key': u'nvme0n1'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/nvme0n1 bs=1M count=110", "delta": "0:00:00.037017", "end": "2019-12-21 15:42:10.356865", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_4", "vg_nvme-lv_1", "vg_nvme-lv_3", "vg_nvme-lv_5", "vg_nvme-lv_2"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-M3E1yd-4qLJ-BJqp-tClY-HPKa-i540-fctM4E", "nvme-INTEL_SSDPEDMD400G4_CVFT6233005M400BGN", "nvme-nvme.8086-43564654363233333030354d34303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2019-12-21 15:42:10.319848", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 11 2019-12-21T15:42:10.389 INFO:teuthology.task.ansible.out:0 MiB) copied, 0.0344113 s, 3.4 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.0344113 s, 3.4 GB/s"], "stdout": "", "stdout_lines": []} 2019-12-21T15:42:10.730 INFO:teuthology.task.ansible.out:changed: [smithi172.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_1', u'dm-uuid-LVM-y1halPOkiUJG3Hdv0eQmk8A7QQ3CY3FDESIRk22UYsDUIFVqAa2SokNTWdtX5mFq'], u'uuids': [u'03dcd4a9-d55d-4b83-b445-78f7dcda1ae0']}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-4'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-4 bs=1M count=110", "delta": "0:00:00.154217", "end": "2019-12-21 15:42:10.705312", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-y1halPOkiUJG3Hdv0eQmk8A7QQ3CY3FDESIRk22UYsDUIFVqAa2SokNTWdtX5mFq"], "labels": [], "masters": [], "uuids": ["03dcd4a9-d55d-4b83-b445-78f7dcda1ae0"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2019-12-21 15:42:10.551095", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.151621 s, 761 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.151621 s, 761 MB/s"], "stdout": "", "stdout_lines": []} 2019-12-21T15:42:11.065 INFO:teuthology.task.ansible.out:changed: [smithi172.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_3', u'dm-uuid-LVM-y1halPOkiUJG3Hdv0eQmk8A7QQ3CY3FDOE8dcEVtfkYOgIB4g3fY71GKn6erNGsx'], u'uuids': [u'06899635-f942-44a1-b2b1-50517f6aaa3b']}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-2'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-2 bs=1M count=110", "delta": "0:00:00.137328", "end": "2019-12-21 15:42:11.036312", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-y1halPOkiUJG3Hdv0eQmk8A7QQ3CY3FDOE8dcEVtfkYOgIB4g3fY71GKn6erNGsx"], "labels": [], "masters": [], "uuids": ["06899635-f942-44a1-b2b1-50517f6aaa3b"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2019-12-21 15:42:10.898984", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.134767 s, 856 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.134767 s, 856 MB/s"], "stdout": "", "stdout_lines": []} 2019-12-21T15:42:11.396 INFO:teuthology.task.ansible.out:changed: [smithi172.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_2', u'dm-uuid-LVM-y1halPOkiUJG3Hdv0eQmk8A7QQ3CY3FDbU33qKUhSCW82PUb6Js6s5Pby0QFWUtP'], u'uuids': [u'a008c94a-a8eb-4e67-8453-e60d99b1ef45']}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-3'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-3 bs=1M count=110", "delta": "0:00:00.140858", "end": "2019-12-21 15:42:11.365660", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-y1halPOkiUJG3Hdv0eQmk8A7QQ3CY3FDbU33qKUhSCW82PUb6Js6s5Pby0QFWUtP"], "labels": [], "masters": [], "uuids": ["a008c94a-a8eb-4e67-8453-e60d99b1ef45"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2019-12-21 15:42:11.224802", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.138163 s, 835 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.138163 s, 835 MB/s"], "stdout": "", "stdout_lines": []} 2019-12-21T15:42:11.717 INFO:teuthology.task.ansible.out:changed: [smithi172.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_5', u'dm-uuid-LVM-y1halPOkiUJG3Hdv0eQmk8A7QQ3CY3FDd1dgZEESwTcW0KdykZgyoZBQvapHhOa0'], u'uuids': [u'e00a3f60-423b-4ccd-a694-e919fa9ef997']}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'14.90 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'31252480', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-0'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-0 bs=1M count=110", "delta": "0:00:00.140637", "end": "2019-12-21 15:42:11.689460", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-y1halPOkiUJG3Hdv0eQmk8A7QQ3CY3FDd1dgZEESwTcW0KdykZgyoZBQvapHhOa0"], "labels": [], "masters": [], "uuids": ["e00a3f60-423b-4ccd-a694-e919fa9ef997"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2019-12-21 15:42:11.548823", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.138111 s, 835 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.138111 s, 835 MB/s"], "stdout": "", "stdout_lines": []} 2019-12-21T15:42:12.036 INFO:teuthology.task.ansible.out:changed: [smithi172.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_4', u'dm-uuid-LVM-y1halPOkiUJG3Hdv0eQmk8A7QQ3CY3FDNmLCTkMonvtl9l3u6cs0sUW0DT7hTrRT'], u'uuids': [u'a04fd9b3-1412-4b03-b7c5-74c7eb4b57d2']}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-1'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-1 bs=1M count=110", "delta": "0:00:00.138388", "end": "2019-12-21 15:42:12.009110", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-y1halPOkiUJG3Hdv0eQmk8A7QQ3CY3FDNmLCTkMonvtl9l3u6cs0sUW0DT7hTrRT"], "labels": [], "masters": [], "uuids": ["a04fd9b3-1412-4b03-b7c5-74c7eb4b57d2"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2019-12-21 15:42:11.870722", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.135754 s, 850 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.135754 s, 850 MB/s"], "stdout": "", "stdout_lines": []} 2019-12-21T15:42:12.085 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:42:12 +0000 (0:00:01.974) 0:03:50.092 ***** 2019-12-21T15:42:12.346 INFO:teuthology.task.ansible.out: TASK [testnode : Remove all LVM data] ****************************************** 2019-12-21T15:42:12.348 INFO:teuthology.task.ansible.out:changed: [smithi172.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "cmd": "dmsetup remove_all --force", "delta": "0:00:00.070872", "end": "2019-12-21 15:42:12.322094", "rc": 0, "start": "2019-12-21 15:42:12.251222", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2019-12-21T15:42:12.388 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:42:12 +0000 (0:00:00.303) 0:03:50.396 ***** 2019-12-21T15:42:12.631 INFO:teuthology.task.ansible.out: TASK [testnode : Check for physical volumes] *********************************** 2019-12-21T15:42:12.632 INFO:teuthology.task.ansible.out:changed: [smithi172.front.sepia.ceph.com] => {"changed": true, "cmd": "pvdisplay | grep 'PV Name' | awk '{ print $3 }'", "delta": "0:00:00.049555", "end": "2019-12-21 15:42:12.609355", "rc": 0, "start": "2019-12-21 15:42:12.559800", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2019-12-21T15:42:12.676 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:42:12 +0000 (0:00:00.287) 0:03:50.683 ***** 2019-12-21T15:42:12.703 INFO:teuthology.task.ansible.out: TASK [testnode : Remove physical volumes] ************************************** 2019-12-21T15:42:12.756 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:42:12 +0000 (0:00:00.080) 0:03:50.764 ***** 2019-12-21T15:42:12.784 INFO:teuthology.task.ansible.out: TASK [testnode : Write a new partition table to {{ item.value.device }}] ******* 2019-12-21T15:42:12.786 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-21T15:42:12.826 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:42:12 +0000 (0:00:00.069) 0:03:50.834 ***** 2019-12-21T15:42:12.863 INFO:teuthology.task.ansible.out: TASK [testnode : Write partition entries to {{ item.0.device }}] *************** 2019-12-21T15:42:12.864 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-21T15:42:12.903 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:42:12 +0000 (0:00:00.076) 0:03:50.910 ***** 2019-12-21T15:42:12.945 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2019-12-21T15:42:12.946 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-21T15:42:12.984 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:42:12 +0000 (0:00:00.080) 0:03:50.991 ***** 2019-12-21T15:42:13.010 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs for {{ item.0.device }}] ****************** 2019-12-21T15:42:13.012 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-21T15:42:13.053 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:42:13 +0000 (0:00:00.069) 0:03:51.061 ***** 2019-12-21T15:42:13.246 INFO:teuthology.task.ansible.out: TASK [testnode : Edit lvm.conf to support LVM on kRBD.] ************************ 2019-12-21T15:42:13.246 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2019-12-21T15:42:13.295 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:42:13 +0000 (0:00:00.241) 0:03:51.302 ***** 2019-12-21T15:42:13.335 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2019-12-21T15:42:13.338 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={u'block_used': 12836194, u'uuid': u'cf568be9-1a39-43d2-a003-fbe6239693bc', u'size_total': 983429562368, u'block_total': 240095108, u'mount': u'/', u'block_available': 227258914, u'size_available': 930852511744, u'fstype': u'ext4', u'inode_total': 61054976, u'inode_available': 60993208, u'device': u'/dev/sda1', u'inode_used': 61768, u'block_size': 4096, u'options': u'rw,seclabel,relatime'}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 227258914, "block_size": 4096, "block_total": 240095108, "block_used": 12836194, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60993208, "inode_total": 61054976, "inode_used": 61768, "mount": "/", "options": "rw,seclabel,relatime", "size_available": 930852511744, "size_total": 983429562368, "uuid": "cf568be9-1a39-43d2-a003-fbe6239693bc"}, "skip_reason": "Conditional result was False"} 2019-12-21T15:42:13.377 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:42:13 +0000 (0:00:00.082) 0:03:51.385 ***** 2019-12-21T15:42:13.420 INFO:teuthology.task.ansible.out: TASK [testnode : Combine list of non-root disks] ******************************* skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-21T15:42:13.456 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:42:13 +0000 (0:00:00.078) 0:03:51.463 ***** 2019-12-21T15:42:13.502 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2019-12-21T15:42:13.504 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-21T15:42:13.549 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:42:13 +0000 (0:00:00.093) 0:03:51.557 ***** 2019-12-21T15:42:13.576 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-21T15:42:13.615 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:42:13 +0000 (0:00:00.066) 0:03:51.623 ***** 2019-12-21T15:42:13.642 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume_groups dict] ************************************ 2019-12-21T15:42:13.644 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-21T15:42:13.682 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:42:13 +0000 (0:00:00.066) 0:03:51.689 ***** 2019-12-21T15:42:13.719 INFO:teuthology.task.ansible.out: TASK [testnode : Determine desired logical volume percentage size] ************* 2019-12-21T15:42:13.720 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-21T15:42:13.755 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:42:13 +0000 (0:00:00.073) 0:03:51.763 ***** 2019-12-21T15:42:13.781 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical_volumes dict] ********************************** 2019-12-21T15:42:13.783 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-21T15:42:13.819 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:42:13 +0000 (0:00:00.063) 0:03:51.827 ***** 2019-12-21T15:42:14.543 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume group(s)] *************************************** 2019-12-21T15:42:14.544 INFO:teuthology.task.ansible.out:changed: [smithi172.front.sepia.ceph.com] => (item={'value': {u'pvs': u'/dev/nvme0n1'}, 'key': u'vg_nvme'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "vg_nvme", "value": {"pvs": "/dev/nvme0n1"}}} 2019-12-21T15:42:14.584 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:42:14 +0000 (0:00:00.764) 0:03:52.591 ***** 2019-12-21T15:42:15.230 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* 2019-12-21T15:42:15.232 INFO:teuthology.task.ansible.out:changed: [smithi172.front.sepia.ceph.com] => (item={'value': {u'vg': u'vg_nvme', u'size': u'14.9G'}, 'key': u'lv_5'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "msg": ""} 2019-12-21T15:42:15.588 INFO:teuthology.task.ansible.out:changed: [smithi172.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_4'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2019-12-21T15:42:15.939 INFO:teuthology.task.ansible.out:changed: [smithi172.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_3'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2019-12-21T15:42:16.271 INFO:teuthology.task.ansible.out:changed: [smithi172.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_2'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2019-12-21T15:42:16.618 INFO:teuthology.task.ansible.out:changed: [smithi172.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_1'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2019-12-21T15:42:16.655 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:42:16 +0000 (0:00:02.071) 0:03:54.663 ***** 2019-12-21T15:42:16.857 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2019-12-21T15:42:16.859 INFO:teuthology.task.ansible.out:changed: [smithi172.front.sepia.ceph.com] => {"changed": true, "path": "/scratch_devs", "state": "absent"} 2019-12-21T15:42:16.896 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:42:16 +0000 (0:00:00.240) 0:03:54.903 ***** 2019-12-21T15:42:16.934 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs] ****************************************** 2019-12-21T15:42:16.935 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={'value': {u'vg': u'vg_nvme', u'size': u'14.9G'}, 'key': u'lv_5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "skip_reason": "Conditional result was False"} 2019-12-21T15:42:17.122 INFO:teuthology.task.ansible.out:changed: [smithi172.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_4'}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2019-12-21T15:42:17.316 INFO:teuthology.task.ansible.out:changed: [smithi172.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_3'}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2019-12-21T15:42:17.491 INFO:teuthology.task.ansible.out:changed: [smithi172.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_2'}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2019-12-21T15:42:17.671 INFO:teuthology.task.ansible.out:changed: [smithi172.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_1'}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2019-12-21T15:42:17.718 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:42:17 +0000 (0:00:00.822) 0:03:55.726 ***** 2019-12-21T15:42:17.917 INFO:teuthology.task.ansible.out: TASK [testnode : Create /var/lib/ceph] ***************************************** 2019-12-21T15:42:17.918 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/var/lib/ceph", "secontext": "unconfined_u:object_r:var_lib_t:s0", "size": 4096, "state": "directory", "uid": 0} 2019-12-21T15:42:17.956 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:42:17 +0000 (0:00:00.237) 0:03:55.963 ***** 2019-12-21T15:42:17.984 INFO:teuthology.task.ansible.out: TASK [testnode : Set xfs_opts on newer OSes] *********************************** 2019-12-21T15:42:17.985 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-21T15:42:18.028 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:42:18 +0000 (0:00:00.071) 0:03:56.035 ***** 2019-12-21T15:42:19.606 INFO:teuthology.task.ansible.out: TASK [testnode : Create xfs filesystem on /dev/vg_nvme/lv_5] ******************* 2019-12-21T15:42:19.608 INFO:teuthology.task.ansible.out:changed: [smithi172.front.sepia.ceph.com] => {"changed": true} 2019-12-21T15:42:19.653 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:42:19 +0000 (0:00:01.625) 0:03:57.660 ***** 2019-12-21T15:42:20.244 INFO:teuthology.task.ansible.out: TASK [testnode : Mount /dev/vg_nvme/lv_5 to /var/lib/ceph] ********************* 2019-12-21T15:42:20.246 INFO:teuthology.task.ansible.out:changed: [smithi172.front.sepia.ceph.com] => {"changed": true, "dump": "0", "fstab": "/etc/fstab", "fstype": "xfs", "name": "/var/lib/ceph", "opts": "defaults,nofail", "passno": "0", "src": "/dev/vg_nvme/lv_5"} 2019-12-21T15:42:20.282 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:42:20 +0000 (0:00:00.629) 0:03:58.290 ***** 2019-12-21T15:42:20.303 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the teuthology user.] *********** 2019-12-21T15:42:20.304 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-21T15:42:20.342 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:42:20 +0000 (0:00:00.059) 0:03:58.349 ***** 2019-12-21T15:42:20.368 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the root user.] ***************** 2019-12-21T15:42:20.370 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-21T15:42:20.413 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:42:20 +0000 (0:00:00.071) 0:03:58.421 ***** 2019-12-21T15:42:20.442 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for the teuthology user.] ****************** 2019-12-21T15:42:20.444 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-21T15:42:20.480 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:42:20 +0000 (0:00:00.066) 0:03:58.487 ***** 2019-12-21T15:42:20.511 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for root.] ********************************* 2019-12-21T15:42:20.512 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-21T15:42:20.552 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:42:20 +0000 (0:00:00.072) 0:03:58.560 ***** 2019-12-21T15:42:20.589 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure perl-doc is installed on apt systems.] ***************** 2019-12-21T15:42:20.590 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-21T15:42:20.626 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:42:20 +0000 (0:00:00.073) 0:03:58.634 ***** 2019-12-21T15:42:20.652 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if Amazon::S3 is installed.] ********************* 2019-12-21T15:42:20.654 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-21T15:42:20.690 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:42:20 +0000 (0:00:00.063) 0:03:58.697 ***** 2019-12-21T15:42:20.726 INFO:teuthology.task.ansible.out: TASK [testnode : Install Amazon::S3.] ****************************************** 2019-12-21T15:42:20.727 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-21T15:42:20.762 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:42:20 +0000 (0:00:00.072) 0:03:58.770 ***** 2019-12-21T15:42:20.787 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on rpm based systems.] ******************** 2019-12-21T15:42:20.789 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-21T15:42:20.824 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:42:20 +0000 (0:00:00.061) 0:03:58.832 ***** 2019-12-21T15:42:20.849 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on apt based systems.] ******************** 2019-12-21T15:42:20.851 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-21T15:42:20.886 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:42:20 +0000 (0:00:00.061) 0:03:58.893 ***** 2019-12-21T15:42:20.911 INFO:teuthology.task.ansible.out: TASK [testnode : Remove racy /etc/network/if-up.d/ntpdate on xenial] *********** 2019-12-21T15:42:20.913 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-21T15:42:20.948 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:42:20 +0000 (0:00:00.062) 0:03:58.955 ***** 2019-12-21T15:42:20.973 INFO:teuthology.task.ansible.out: TASK [testnode : Create the ntp.conf file.] ************************************ 2019-12-21T15:42:20.975 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-21T15:42:21.010 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:42:21 +0000 (0:00:00.061) 0:03:59.017 ***** 2019-12-21T15:42:21.491 INFO:teuthology.task.ansible.out: TASK [testnode : Create the chrony.conf file] ********************************** 2019-12-21T15:42:21.491 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => {"changed": false, "checksum": "a7ecef00073feab34f868065d34e9dac55134241", "dest": "/etc/chrony.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/chrony.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 215, "state": "file", "uid": 0} 2019-12-21T15:42:21.536 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:42:21 +0000 (0:00:00.526) 0:03:59.543 ***** 2019-12-21T15:42:21.845 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ntpd is running.] *********************************** 2019-12-21T15:42:21.847 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "chronyd", "state": "started", "status": {"ActiveEnterTimestamp": "Sat 2019-12-21 15:37:33 UTC", "ActiveEnterTimestampMonotonic": "13404232", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "tmp.mount ntpdate.service system.slice systemd-journald.socket basic.target sysinit.target sntp.service systemd-tmpfiles-setup.service -.mount ntpd.service", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Sat 2019-12-21 15:37:31 UTC", "AssertTimestampMonotonic": "12330523", "Before": "shutdown.target multi-user.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "no", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Sat 2019-12-21 15:37:31 UTC", "ConditionTimestampMonotonic": "12330442", "ConfigurationDirectoryMode": "0755", "Conflicts": "systemd-timesyncd.service ntpd.service shutdown.target", "ControlGroup": "/system.slice/chronyd.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description" 2019-12-21T15:42:21.848 INFO:teuthology.task.ansible.out:: "NTP client/server", "DevicePolicy": "auto", "Documentation": "man:chronyd(8) man:chrony.conf(5)", "DynamicUser": "no", "EnvironmentFiles": "/etc/sysconfig/chronyd (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "787", "ExecMainStartTimestamp": "Sat 2019-12-21 15:37:32 UTC", "ExecMainStartTimestampMonotonic": "13356713", "ExecMainStatus": "0", "ExecStart": "{ path=/usr/sbin/chronyd ; argv[]=/usr/sbin/chronyd $OPTIONS ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStartPost": "{ path=/usr/libexec/chrony-helper ; argv[]=/usr/libexec/chrony-helper update-daemon ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/chronyd.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "chronyd.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Sat 2019-12-21 15:37:31 UTC", "InactiveExitTimestampMonotonic": "12340280", "InvocationID": "aaeeb3abfbe44a659c547acfff49451c", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "infinity", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "1677721 2019-12-21T15:42:21.850 INFO:teuthology.task.ansible.out:6", "LimitMEMLOCKSoft": "16777216", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "127463", "LimitNPROCSoft": "127463", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127463", "LimitSIGPENDINGSoft": "127463", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "787", "MemoryAccounting": "yes", "MemoryCurrent": "2727936", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "chronyd.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PIDFile": "/var/run/chrony/chronyd.pid", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateMounts": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "yes", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "full", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "sysinit.target system.slice -.mount", "RequiresMountsFor": "/var/tmp", "Restart": "no", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "Standar 2019-12-21T15:42:21.851 INFO:teuthology.task.ansible.out:dError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Sat 2019-12-21 15:37:33 UTC", "StateChangeTimestampMonotonic": "13404232", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "1", "TasksMax": "26213", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "forking", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Sat 2019-12-21 15:37:32 UTC", "WatchdogTimestampMonotonic": "13356727", "WatchdogUSec": "0"}} 2019-12-21T15:42:21.890 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:42:21 +0000 (0:00:00.353) 0:03:59.897 ***** 2019-12-21T15:42:21.917 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on rpm based systems.] ********************* 2019-12-21T15:42:21.919 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-21T15:42:21.955 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:42:21 +0000 (0:00:00.065) 0:03:59.962 ***** 2019-12-21T15:42:21.985 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on apt based systems.] ********************* 2019-12-21T15:42:21.987 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-21T15:42:22.023 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:42:22 +0000 (0:00:00.068) 0:04:00.031 ***** 2019-12-21T15:42:22.052 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on zypper based systems.] ****************** 2019-12-21T15:42:22.053 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-21T15:42:22.093 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:42:22 +0000 (0:00:00.070) 0:04:00.101 ***** 2019-12-21T15:42:22.293 INFO:teuthology.task.ansible.out: TASK [testnode : Create the .pip directory for the teuthology user.] *********** 2019-12-21T15:42:22.295 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => {"changed": false, "gid": 1205, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 4096, "state": "directory", "uid": 1000} 2019-12-21T15:42:22.335 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:42:22 +0000 (0:00:00.242) 0:04:00.343 ***** 2019-12-21T15:42:22.814 INFO:teuthology.task.ansible.out: TASK [testnode : Create pip.conf and configure it to use our mirror] *********** 2019-12-21T15:42:22.814 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => {"changed": false, "checksum": "959d37ab30545da205b389a6b7561b225a478a06", "dest": "/home/ubuntu/.pip/pip.conf", "gid": 1205, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip/pip.conf", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 73, "state": "file", "uid": 1000} 2019-12-21T15:42:22.854 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:42:22 +0000 (0:00:00.518) 0:04:00.862 ***** 2019-12-21T15:42:24.235 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages via pip] ************************************* 2019-12-21T15:42:24.236 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => {"changed": false, "cmd": ["/bin/pip3", "install", "remoto>=0.0.35"], "name": ["remoto>=0.0.35"], "requirements": null, "state": "present", "stderr": "WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead.\n", "stderr_lines": ["WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead."], "stdout": "Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python3.6/site-packages\nRequirement already satisfied: execnet in /usr/local/lib/python3.6/site-packages (from remoto>=0.0.35)\nRequirement already satisfied: apipkg>=1.4 in /usr/local/lib/python3.6/site-packages (from execnet->remoto>=0.0.35)\n", "stdout_lines": ["Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python3.6/site-packages", "Requirement already satisfied: execnet in /usr/local/lib/python3.6/site-packages (from remoto>=0.0.35)", "Requirement already satisfied: apipkg>=1.4 in /usr/local/lib/python3.6/site-packages (from execnet->remoto>=0.0.35)"], "version": null, "virtualenv": null} 2019-12-21T15:42:24.283 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:42:24 +0000 (0:00:01.429) 0:04:02.291 ***** 2019-12-21T15:42:24.311 INFO:teuthology.task.ansible.out: TASK [testnode : Purge resolvconf] ********************************************* 2019-12-21T15:42:24.312 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-21T15:42:24.351 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:42:24 +0000 (0:00:00.067) 0:04:02.358 ***** 2019-12-21T15:42:24.377 INFO:teuthology.task.ansible.out: TASK [testnode : Set interface] ************************************************ 2019-12-21T15:42:24.378 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => {"ansible_facts": {"interface": "enp3s0f1"}, "changed": false} 2019-12-21T15:42:24.418 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:42:24 +0000 (0:00:00.067) 0:04:02.426 ***** 2019-12-21T15:42:24.613 INFO:teuthology.task.ansible.out: TASK [testnode : Check for presence of /etc/network/interfaces] **************** 2019-12-21T15:42:24.615 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => {"changed": false, "stat": {"exists": false}} 2019-12-21T15:42:24.653 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:42:24 +0000 (0:00:00.234) 0:04:02.661 ***** 2019-12-21T15:42:24.681 INFO:teuthology.task.ansible.out: TASK [testnode : Rewrite /etc/network/interfaces to use dhcp] ****************** 2019-12-21T15:42:24.683 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-21T15:42:24.723 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:42:24 +0000 (0:00:00.069) 0:04:02.731 ***** 2019-12-21T15:42:24.757 INFO:teuthology.task.ansible.out: 2019-12-21T15:42:24.759 INFO:teuthology.task.ansible.out:TASK [testnode : Set bounce_interface if we just enabled dhcp] ***************** ok: [smithi172.front.sepia.ceph.com] => {"ansible_facts": {"bounce_interface": false}, "changed": false} 2019-12-21T15:42:24.808 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:42:24 +0000 (0:00:00.084) 0:04:02.815 ***** 2019-12-21T15:42:24.810 INFO:teuthology.task.ansible.out: 2019-12-21T15:42:24.835 INFO:teuthology.task.ansible.out: TASK [testnode : ifdown and ifup] ********************************************** 2019-12-21T15:42:24.837 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-21T15:42:24.874 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:42:24 +0000 (0:00:00.067) 0:04:02.882 ***** 2019-12-21T15:42:25.081 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure lab_domain is in search domains in /etc/resolv.conf] *** 2019-12-21T15:42:25.082 INFO:teuthology.task.ansible.out:changed: [smithi172.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line replaced"} 2019-12-21T15:42:25.127 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:42:25 +0000 (0:00:00.252) 0:04:03.134 ***** 2019-12-21T15:42:25.348 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure domain is set in /etc/resolv.conf] ********************* 2019-12-21T15:42:25.350 INFO:teuthology.task.ansible.out:changed: [smithi172.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line added"} 2019-12-21T15:42:25.395 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:42:25 +0000 (0:00:00.267) 0:04:03.402 ***** 2019-12-21T15:42:25.423 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ethtool is installed] ******************************* 2019-12-21T15:42:25.424 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-21T15:42:25.461 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:42:25 +0000 (0:00:00.066) 0:04:03.469 ***** 2019-12-21T15:42:25.498 INFO:teuthology.task.ansible.out: TASK [testnode : grep ethtool for secondary NIC MAC address] ******************* 2019-12-21T15:42:25.499 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item=eno1) => {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False"} 2019-12-21T15:42:25.500 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item=enp3s0f1) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False"} 2019-12-21T15:42:25.510 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item=eno2) => {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False"} 2019-12-21T15:42:25.511 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item=lo) => {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False"} 2019-12-21T15:42:25.522 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item=enp3s0f0) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0", "skip_reason": "Conditional result was False"} 2019-12-21T15:42:25.569 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:42:25 +0000 (0:00:00.107) 0:04:03.577 ***** 2019-12-21T15:42:25.613 INFO:teuthology.task.ansible.out: TASK [testnode : Define net_to_configure var] ********************************** 2019-12-21T15:42:25.615 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={'skip_reason': u'Conditional result was False', 'item': u'eno1', 'skipped': True, 'ansible_loop_var': u'item', 'changed': False}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} skipping: [smithi172.front.sepia.ceph.com] => (item={'skip_reason': u'Conditional result was False', 'item': u'enp3s0f1', 'skipped': True, 'ansible_loop_var': u'item', 'changed': False}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2019-12-21T15:42:25.626 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={'skip_reason': u'Conditional result was False', 'item': u'eno2', 'skipped': True, 'ansible_loop_var': u'item', 'changed': False}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2019-12-21T15:42:25.627 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={'skip_reason': u'Conditional result was False', 'item': u'lo', 'skipped': True, 'ansible_loop_var': u'item', 'changed': False}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2019-12-21T15:42:25.642 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={'skip_reason': u'Conditional result was False', 'item': u'enp3s0f0', 'skipped': True, 'ansible_loop_var': u'item', 'changed': False}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2019-12-21T15:42:25.686 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:42:25 +0000 (0:00:00.116) 0:04:03.693 ***** 2019-12-21T15:42:25.716 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2019-12-21T15:42:25.717 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-21T15:42:25.774 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:42:25 +0000 (0:00:00.088) 0:04:03.782 ***** 2019-12-21T15:42:25.800 INFO:teuthology.task.ansible.out: TASK [testnode : Check if {{ nic_to_configure }} is 10Gb] ********************** 2019-12-21T15:42:25.802 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-21T15:42:25.840 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:42:25 +0000 (0:00:00.066) 0:04:03.848 ***** 2019-12-21T15:42:25.866 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU to 9000 if 10Gb] ************************************** 2019-12-21T15:42:25.868 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-21T15:42:25.915 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:42:25 +0000 (0:00:00.074) 0:04:03.922 ***** 2019-12-21T15:42:25.941 INFO:teuthology.task.ansible.out: TASK [testnode : Write Ubuntu network config for {{ nic_to_configure }}] ******* 2019-12-21T15:42:25.943 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-21T15:42:25.991 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:42:25 +0000 (0:00:00.076) 0:04:03.999 ***** 2019-12-21T15:42:26.017 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU on Ubuntu] ******************************************** 2019-12-21T15:42:26.019 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-21T15:42:26.071 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:42:26 +0000 (0:00:00.079) 0:04:04.079 ***** 2019-12-21T15:42:26.101 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2019-12-21T15:42:26.103 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-21T15:42:26.144 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:42:26 +0000 (0:00:00.073) 0:04:04.152 ***** 2019-12-21T15:42:26.182 INFO:teuthology.task.ansible.out: TASK [testnode : Write RHEL/CentOS network config for {{ nic_to_configure }}] *** 2019-12-21T15:42:26.184 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-21T15:42:26.223 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:42:26 +0000 (0:00:00.078) 0:04:04.231 ***** 2019-12-21T15:42:26.250 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2019-12-21T15:42:26.251 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-21T15:42:26.294 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:42:26 +0000 (0:00:00.070) 0:04:04.301 ***** 2019-12-21T15:42:27.511 INFO:teuthology.task.ansible.out: TASK [testnode : List any leftover Ceph artifacts from previous jobs] ********** 2019-12-21T15:42:27.512 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item=/var/run/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/run/ -name \"*ceph*\"", "delta": "0:00:01.005988", "end": "2019-12-21 15:42:27.478410", "failed_when_result": false, "item": "/var/run/", "rc": 0, "start": "2019-12-21 15:42:26.472422", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2019-12-21T15:42:27.708 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item=/etc/systemd/system/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/systemd/system/ -name \"*ceph*\"", "delta": "0:00:00.003021", "end": "2019-12-21 15:42:27.676601", "failed_when_result": false, "item": "/etc/systemd/system/", "rc": 0, "start": "2019-12-21 15:42:27.673580", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2019-12-21T15:42:27.895 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item=/etc/ceph) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/ceph -name \"*ceph*\"", "delta": "0:00:00.002900", "end": "2019-12-21 15:42:27.863606", "failed_when_result": false, "item": "/etc/ceph", "msg": "non-zero return code", "rc": 1, "start": "2019-12-21 15:42:27.860706", "stderr": "find: ‘/etc/ceph’: No such file or directory", "stderr_lines": ["find: ‘/etc/ceph’: No such file or directory"], "stdout": "", "stdout_lines": []} 2019-12-21T15:42:28.102 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item=/var/log/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/log/ -name \"*ceph*\"", "delta": "0:00:00.022311", "end": "2019-12-21 15:42:28.066221", "failed_when_result": false, "item": "/var/log/", "rc": 0, "start": "2019-12-21 15:42:28.043910", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2019-12-21T15:42:28.144 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:42:28 +0000 (0:00:01.848) 0:04:06.150 ***** 2019-12-21T15:42:28.173 INFO:teuthology.task.ansible.out: TASK [testnode : Delete any leftover Ceph artifacts from previous jobs] ******** 2019-12-21T15:42:28.211 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:42:28 +0000 (0:00:00.068) 0:04:06.218 ***** 2019-12-21T15:42:28.407 INFO:teuthology.task.ansible.out: TASK [testnode : Touch /ceph-qa-ready] ***************************************** 2019-12-21T15:42:28.408 INFO:teuthology.task.ansible.out:changed: [smithi172.front.sepia.ceph.com] => {"changed": true, "dest": "/ceph-qa-ready", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "secontext": "unconfined_u:object_r:etc_runtime_t:s0", "size": 0, "state": "file", "uid": 0} 2019-12-21T15:42:28.467 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:42:28 +0000 (0:00:00.253) 0:04:06.472 ***** 2019-12-21T15:42:28.825 INFO:teuthology.task.ansible.out: RUNNING HANDLER [common : restart nagios-nrpe-server] ************************** 2019-12-21T15:42:28.828 INFO:teuthology.task.ansible.out:changed: [smithi172.front.sepia.ceph.com] => {"changed": true, "name": "nrpe", "state": "started", "status": {"ActiveEnterTimestamp": "Sat 2019-12-21 15:40:39 UTC", "ActiveEnterTimestampMonotonic": "200366458", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "systemd-journald.socket systemd-tmpfiles-setup.service local-fs.target sysinit.target tmp.mount network.target time-sync.target -.mount nss-lookup.target var-run.mount basic.target system.slice", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Sat 2019-12-21 15:40:39 UTC", "AssertTimestampMonotonic": "200365610", "Before": "xdm.service shutdown.target getty@tty1.service multi-user.target plymouth-quit.service", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Sat 2019-12-21 15:40:39 UTC", "ConditionTimestampMonotonic": "200365609", "ConfigurationDirectoryMode": "0755", "Conflicts": "nrpe.socket shutdown.target", "ControlGroup": "/system.slice/nrpe.service", "ControlPID": "0", "DefaultDepe 2019-12-21T15:42:28.829 INFO:teuthology.task.ansible.out:ndencies": "yes", "Delegate": "no", "Description": "Nagios Remote Plugin Executor", "DevicePolicy": "auto", "Documentation": "http://www.nagios.org/documentation", "DynamicUser": "no", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "16507", "ExecMainStartTimestamp": "Sat 2019-12-21 15:40:39 UTC", "ExecMainStartTimestampMonotonic": "200366428", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/nrpe ; argv[]=/usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -f ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStopPost": "{ path=/bin/rm ; argv[]=/bin/rm -f /var/run/nrpe/nrpe.pid ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/nrpe.service", "GID": "985", "Group": "nrpe", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "nrpe.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Sat 2019-12-21 15:40:39 UTC", "InactiveExitTimestampMonotonic": "200366458", "InvocationID": "dc0ace1767fb437a80215c496782b708", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "infinity", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft 2019-12-21T15:42:28.831 INFO:teuthology.task.ansible.out:": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "16777216", "LimitMEMLOCKSoft": "16777216", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "127463", "LimitNPROCSoft": "127463", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127463", "LimitSIGPENDINGSoft": "127463", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "16507", "MemoryAccounting": "yes", "MemoryCurrent": "712704", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "nrpe.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "-500", "OnFailureJobMode": "replace", "PIDFile": "/var/run/nrpe/nrpe.pid", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateMounts": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "-.mount sysinit.target system.slice", "RequiresMountsFor": "/run/nrpe /var/tmp", "Restart": "on-abort", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectory": "nrpe", "RuntimeDirectoryMode": "075 2019-12-21T15:42:28.832 INFO:teuthology.task.ansible.out:5", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Sat 2019-12-21 15:40:39 UTC", "StateChangeTimestampMonotonic": "200366458", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "1", "TasksMax": "26213", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min", "TimerSlackNSec": "50000", "Transient": "no", "Type": "simple", "UID": "989", "UMask": "0022", "UnitFilePreset": "disabled", "UnitFileState": "enabled", "User": "nrpe", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Sat 2019-12-21 15:40:39 UTC", "WatchdogTimestampMonotonic": "200366458", "WatchdogUSec": "0"}} 2019-12-21T15:42:28.895 INFO:teuthology.task.ansible.out: PLAY [cobbler] ***************************************************************** skipping: no hosts matched 2019-12-21T15:42:28.898 INFO:teuthology.task.ansible.out: PLAY [paddles] ***************************************************************** 2019-12-21T15:42:28.899 INFO:teuthology.task.ansible.out:skipping: no hosts matched 2019-12-21T15:42:28.901 INFO:teuthology.task.ansible.out: PLAY [pulpito] ***************************************************************** skipping: no hosts matched PLAY RECAP ********************************************************************* smithi172.front.sepia.ceph.com : ok=130 changed=54 unreachable=0 failed=0 skipped=153 rescued=0 ignored=4 2019-12-21T15:42:28.901 INFO:teuthology.task.ansible.out:Saturday 21 December 2019 15:42:28 +0000 (0:00:00.436) 0:04:06.908 ***** =============================================================================== 2019-12-21T15:42:28.902 INFO:teuthology.task.ansible.out:users : Update authorized_keys using the keys repo --------------------- 29.82s users : Create all admin users with sudo access. ----------------------- 23.11s testnode : Install packages -------------------------------------------- 19.97s common : nrpe - Load SELinux policy package ---------------------------- 16.12s testnode : Ensure ceph packages are not present. ------------------------ 9.98s testnode : Zap all non-root disks --------------------------------------- 8.98s common : Install nrpe package and dependencies (RHEL/CentOS) ------------ 8.96s users : Remove revoked users -------------------------------------------- 8.16s common : Enable necessary rhsm repos ------------------------------------ 7.90s common : Register with subscription-manager. ---------------------------- 7.52s common : Get list of enabled RHSM repos --------------------------------- 6.72s common : Update yum cache ----------------------------------------------- 5.08s testnode : Ensure ceph dependency packages are not present. ------------- 4.98s common : Disable all rhsm repos ----------------------------------------- 4.46s common : Install Kerberos Packages (RedHat) ----------------------------- 3.17s Gathering Facts --------------------------------------------------------- 2.65s ansible-managed : Add authorized keys for the ansible user. ------------- 2.51s common : nrpe - Set SELinux boolean nagios_run_sudo true ---------------- 2.15s 2019-12-21T15:42:28.902 INFO:teuthology.task.ansible.out:users : Update authorized_keys for each user with literal keys ---------- 2.11s testnode : Create logical volume(s) ------------------------------------- 2.07s 2019-12-21T15:42:29.186 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi172.front.sepia.ceph.com')] 2019-12-21T15:42:29.188 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2019-12-21T15:42:29.189 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi172.front.sepia.ceph.com', 'timeout': 60} 2019-12-21T15:42:29.258 INFO:teuthology.orchestra.run.smithi172:> true 2019-12-21T15:42:29.343 INFO:teuthology.run_tasks:Running task clock... 2019-12-21T15:42:29.357 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2019-12-21T15:42:29.358 INFO:teuthology.orchestra.run.smithi172:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2019-12-21T15:42:29.417 INFO:teuthology.orchestra.run.smithi172.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2019-12-21T15:42:29.435 INFO:teuthology.orchestra.run.smithi172.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2019-12-21T15:42:29.478 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: ntpd: command not found 2019-12-21T15:42:29.491 INFO:teuthology.orchestra.run.smithi172.stdout:506 Cannot talk to daemon 2019-12-21T15:42:29.508 INFO:teuthology.orchestra.run.smithi172.stderr:Failed to start ntp.service: Unit ntp.service not found. 2019-12-21T15:42:29.526 INFO:teuthology.orchestra.run.smithi172.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2019-12-21T15:42:29.584 INFO:teuthology.orchestra.run.smithi172.stderr:bash: ntpq: command not found 2019-12-21T15:42:29.586 INFO:teuthology.orchestra.run.smithi172.stdout:210 Number of sources = 4 2019-12-21T15:42:29.586 INFO:teuthology.orchestra.run.smithi172.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2019-12-21T15:42:29.587 INFO:teuthology.orchestra.run.smithi172.stdout:=============================================================================== 2019-12-21T15:42:29.587 INFO:teuthology.orchestra.run.smithi172.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2019-12-21T15:42:29.587 INFO:teuthology.orchestra.run.smithi172.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2019-12-21T15:42:29.587 INFO:teuthology.orchestra.run.smithi172.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2019-12-21T15:42:29.587 INFO:teuthology.orchestra.run.smithi172.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2019-12-21T15:42:29.588 INFO:teuthology.run_tasks:Running task install... 2019-12-21T15:42:29.601 DEBUG:teuthology.task.install:project ceph 2019-12-21T15:42:29.601 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'sha1': 'c6e9f8f9e7516809792ffb509ef4a897f5c2f45e'}} 2019-12-21T15:42:29.601 DEBUG:teuthology.task.install:config {'sha1': 'c6e9f8f9e7516809792ffb509ef4a897f5c2f45e'} 2019-12-21T15:42:29.601 INFO:teuthology.task.install:Using flavor: basic 2019-12-21T15:42:29.608 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-cloud', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2019-12-21T15:42:29.608 INFO:teuthology.task.install:extra packages: [] 2019-12-21T15:42:29.609 INFO:teuthology.orchestra.run.smithi172:> sudo lsb_release -is 2019-12-21T15:42:29.665 INFO:teuthology.orchestra.run.smithi172.stdout:RedHatEnterprise 2019-12-21T15:42:29.666 DEBUG:teuthology.misc:System to be installed: RedHatEnterprise 2019-12-21T15:42:29.666 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'sha1': 'c6e9f8f9e7516809792ffb509ef4a897f5c2f45e', 'exclude_packages': [], 'debuginfo': None, 'install_ceph_packages': True, 'wait_for_package': False, 'downgrade_packages': [], 'project': 'ceph', 'tag': None, 'extra_packages': [], 'branch': None, 'packages': {}, 'repos_only': False, 'extra_system_packages': [], 'flavor': 'basic', 'extras': None} 2019-12-21T15:42:29.667 INFO:teuthology.orchestra.run.smithi172:> sudo lsb_release -is 2019-12-21T15:42:29.742 INFO:teuthology.orchestra.run.smithi172.stdout:RedHatEnterprise 2019-12-21T15:42:29.743 DEBUG:teuthology.misc:System to be installed: RedHatEnterprise 2019-12-21T15:42:29.743 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=c6e9f8f9e7516809792ffb509ef4a897f5c2f45e 2019-12-21T15:42:29.896 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-sage-testing-2019-12-20-2043/c6e9f8f9e7516809792ffb509ef4a897f5c2f45e/centos/8/flavors/default/ 2019-12-21T15:42:29.897 INFO:teuthology.task.install.rpm:Package version is 15.0.0-8795.gc6e9f8f 2019-12-21T15:42:30.033 INFO:teuthology.orchestra.run.smithi172:> sudo sh -c 'cat > /etc/yum.repos.d/ceph.repo' 2019-12-21T15:42:30.081 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-cloud, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2019-12-21T15:42:30.081 INFO:teuthology.orchestra.run.smithi172:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/c6e9f8f9e7516809792ffb509ef4a897f5c2f45e/;g' /etc/yum.repos.d/ceph.repo ; fi 2019-12-21T15:42:30.150 INFO:teuthology.orchestra.run.smithi172:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2019-12-21T15:42:30.230 INFO:teuthology.orchestra.run.smithi172:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2019-12-21T15:42:30.297 INFO:teuthology.orchestra.run.smithi172.stdout:check_obsoletes = 1 2019-12-21T15:42:30.298 INFO:teuthology.orchestra.run.smithi172:> sudo yum clean all 2019-12-21T15:42:30.661 INFO:teuthology.orchestra.run.smithi172.stdout:Updating Subscription Management repositories. 2019-12-21T15:42:31.008 INFO:teuthology.orchestra.run.smithi172.stdout:61 files removed 2019-12-21T15:42:31.057 INFO:teuthology.orchestra.run.smithi172:> sudo yum -y install ceph-radosgw 2019-12-21T15:42:31.379 INFO:teuthology.orchestra.run.smithi172.stdout:Updating Subscription Management repositories. 2019-12-21T15:42:32.068 INFO:teuthology.orchestra.run.smithi172.stdout:ceph packages for x86_64 307 kB/s | 68 kB 00:00 2019-12-21T15:42:32.281 INFO:teuthology.orchestra.run.smithi172.stdout:ceph noarch packages 66 kB/s | 12 kB 00:00 2019-12-21T15:42:32.485 INFO:teuthology.orchestra.run.smithi172.stdout:ceph source packages 7.5 kB/s | 1.4 kB 00:00 2019-12-21T15:42:33.124 INFO:teuthology.orchestra.run.smithi172.stdout:Copr repo for ceph-el8 owned by ktdreyer 56 kB/s | 35 kB 00:00 2019-12-21T15:42:33.951 INFO:teuthology.orchestra.run.smithi172.stdout:Extra Packages for Enterprise Linux 8 - x86_64 5.4 MB/s | 4.3 MB 00:00 2019-12-21T15:42:34.782 INFO:teuthology.orchestra.run.smithi172.stdout:lab-extras 4.8 MB/s | 33 kB 00:00 2019-12-21T15:42:34.971 INFO:teuthology.orchestra.run.smithi172.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 61 MB/s | 9.4 MB 00:00 2019-12-21T15:42:36.410 INFO:teuthology.orchestra.run.smithi172.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 56 MB/s | 8.1 MB 00:00 2019-12-21T15:42:38.603 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2019-12-21T15:42:38.606 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2019-12-21T15:42:38.607 INFO:teuthology.orchestra.run.smithi172.stdout: Package ArchVersion Repository Size 2019-12-21T15:42:38.607 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2019-12-21T15:42:38.607 INFO:teuthology.orchestra.run.smithi172.stdout:Installing: 2019-12-21T15:42:38.607 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-radosgw x86_642:15.0.0-8795.gc6e9f8f.el8 ceph 6.3 M 2019-12-21T15:42:38.607 INFO:teuthology.orchestra.run.smithi172.stdout:Installing dependencies: 2019-12-21T15:42:38.607 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-base x86_642:15.0.0-8795.gc6e9f8f.el8 ceph 5.4 M 2019-12-21T15:42:38.608 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-common x86_642:15.0.0-8795.gc6e9f8f.el8 ceph 20 M 2019-12-21T15:42:38.608 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-selinux x86_642:15.0.0-8795.gc6e9f8f.el8 ceph 25 k 2019-12-21T15:42:38.608 INFO:teuthology.orchestra.run.smithi172.stdout: libcephfs2 x86_642:15.0.0-8795.gc6e9f8f.el8 ceph 504 k 2019-12-21T15:42:38.608 INFO:teuthology.orchestra.run.smithi172.stdout: librados2 x86_642:15.0.0-8795.gc6e9f8f.el8 ceph 3.5 M 2019-12-21T15:42:38.608 INFO:teuthology.orchestra.run.smithi172.stdout: libradosstriper1 x86_642:15.0.0-8795.gc6e9f8f.el8 ceph 377 k 2019-12-21T15:42:38.609 INFO:teuthology.orchestra.run.smithi172.stdout: librbd1 x86_642:15.0.0-8795.gc6e9f8f.el8 ceph 1.9 M 2019-12-21T15:42:38.609 INFO:teuthology.orchestra.run.smithi172.stdout: librgw2 x86_642:15.0.0-8795.gc6e9f8f.el8 ceph 5.6 M 2019-12-21T15:42:38.609 INFO:teuthology.orchestra.run.smithi172.stdout: python3-ceph-argparse x86_642:15.0.0-8795.gc6e9f8f.el8 ceph 40 k 2019-12-21T15:42:38.609 INFO:teuthology.orchestra.run.smithi172.stdout: python3-ceph-common x86_642:15.0.0-8795.gc6e9f8f.el8 ceph 46 k 2019-12-21T15:42:38.609 INFO:teuthology.orchestra.run.smithi172.stdout: python3-cephfs x86_642:15.0.0-8795.gc6e9f8f.el8 ceph 146 k 2019-12-21T15:42:38.609 INFO:teuthology.orchestra.run.smithi172.stdout: python3-rados x86_642:15.0.0-8795.gc6e9f8f.el8 ceph 348 k 2019-12-21T15:42:38.609 INFO:teuthology.orchestra.run.smithi172.stdout: python3-rbd x86_642:15.0.0-8795.gc6e9f8f.el8 ceph 302 k 2019-12-21T15:42:38.609 INFO:teuthology.orchestra.run.smithi172.stdout: python3-rgw x86_642:15.0.0-8795.gc6e9f8f.el8 ceph 113 k 2019-12-21T15:42:38.610 INFO:teuthology.orchestra.run.smithi172.stdout: gperftools-libs x86_642.7-6.el8 epel 315 k 2019-12-21T15:42:38.610 INFO:teuthology.orchestra.run.smithi172.stdout: liboath x86_642.6.2-3.el8 epel 59 k 2019-12-21T15:42:38.610 INFO:teuthology.orchestra.run.smithi172.stdout: libunwind x86_641.3.1-3.el8 epel 75 k 2019-12-21T15:42:38.610 INFO:teuthology.orchestra.run.smithi172.stdout: leveldb x86_641.21-2.el8 lab-extras 179 k 2019-12-21T15:42:38.610 INFO:teuthology.orchestra.run.smithi172.stdout: libbabeltrace x86_641.5.6-2.el8 lab-extras 200 k 2019-12-21T15:42:38.610 INFO:teuthology.orchestra.run.smithi172.stdout: lttng-ust x86_642.8.1-11.el8 lab-extras 257 k 2019-12-21T15:42:38.610 INFO:teuthology.orchestra.run.smithi172.stdout: python3-prettytable noarch0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2019-12-21T15:42:38.610 INFO:teuthology.orchestra.run.smithi172.stdout: librdkafka x86_640.11.4-1.el8 rhel-8-for-x86_64-appstream-rpms 353 k 2019-12-21T15:42:38.611 INFO:teuthology.orchestra.run.smithi172.stdout: rdma-core x86_6422-2.el8 rhel-8-for-x86_64-baseos-rpms 53 k 2019-12-21T15:42:38.611 INFO:teuthology.orchestra.run.smithi172.stdout: librdmacm x86_6422-2.el8 rhel-8-for-x86_64-baseos-rpms 70 k 2019-12-21T15:42:38.611 INFO:teuthology.orchestra.run.smithi172.stdout: userspace-rcu x86_640.10.1-2.el8 rhel-8-for-x86_64-baseos-rpms 101 k 2019-12-21T15:42:38.611 INFO:teuthology.orchestra.run.smithi172.stdout: librabbitmq x86_640.9.0-1.el8 rhel-8-for-x86_64-baseos-rpms 47 k 2019-12-21T15:42:38.611 INFO:teuthology.orchestra.run.smithi172.stdout: libibverbs x86_6422-2.el8 rhel-8-for-x86_64-baseos-rpms 237 k 2019-12-21T15:42:38.611 INFO:teuthology.orchestra.run.smithi172.stdout: 2019-12-21T15:42:38.611 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction Summary 2019-12-21T15:42:38.612 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2019-12-21T15:42:38.612 INFO:teuthology.orchestra.run.smithi172.stdout:Install 28 Packages 2019-12-21T15:42:38.612 INFO:teuthology.orchestra.run.smithi172.stdout: 2019-12-21T15:42:38.613 INFO:teuthology.orchestra.run.smithi172.stdout:Total download size: 46 M 2019-12-21T15:42:38.613 INFO:teuthology.orchestra.run.smithi172.stdout:Installed size: 190 M 2019-12-21T15:42:38.613 INFO:teuthology.orchestra.run.smithi172.stdout:Downloading Packages: 2019-12-21T15:42:39.600 INFO:teuthology.orchestra.run.smithi172.stdout:(1/28): ceph-base-15.0.0-8795.gc6e9f8f.el8.x86_ 7.9 MB/s | 5.4 MB 00:00 2019-12-21T15:42:39.641 INFO:teuthology.orchestra.run.smithi172.stdout:(2/28): ceph-selinux-15.0.0-8795.gc6e9f8f.el8.x 599 kB/s | 25 kB 00:00 2019-12-21T15:42:39.701 INFO:teuthology.orchestra.run.smithi172.stdout:(3/28): libcephfs2-15.0.0-8795.gc6e9f8f.el8.x86 8.3 MB/s | 504 kB 00:00 2019-12-21T15:42:39.722 INFO:teuthology.orchestra.run.smithi172.stdout:(4/28): ceph-radosgw-15.0.0-8795.gc6e9f8f.el8.x 7.8 MB/s | 6.3 MB 00:00 2019-12-21T15:42:39.778 INFO:teuthology.orchestra.run.smithi172.stdout:(5/28): libradosstriper1-15.0.0-8795.gc6e9f8f.e 6.6 MB/s | 377 kB 00:00 2019-12-21T15:42:40.031 INFO:teuthology.orchestra.run.smithi172.stdout:(6/28): librbd1-15.0.0-8795.gc6e9f8f.el8.x86_64 7.3 MB/s | 1.9 MB 00:00 2019-12-21T15:42:40.054 INFO:teuthology.orchestra.run.smithi172.stdout:(7/28): librados2-15.0.0-8795.gc6e9f8f.el8.x86_ 10 MB/s | 3.5 MB 00:00 2019-12-21T15:42:40.107 INFO:teuthology.orchestra.run.smithi172.stdout:(8/28): python3-ceph-argparse-15.0.0-8795.gc6e9 754 kB/s | 40 kB 00:00 2019-12-21T15:42:40.163 INFO:teuthology.orchestra.run.smithi172.stdout:(9/28): python3-ceph-common-15.0.0-8795.gc6e9f8 834 kB/s | 46 kB 00:00 2019-12-21T15:42:40.226 INFO:teuthology.orchestra.run.smithi172.stdout:(10/28): python3-cephfs-15.0.0-8795.gc6e9f8f.el 2.3 MB/s | 146 kB 00:00 2019-12-21T15:42:40.299 INFO:teuthology.orchestra.run.smithi172.stdout:(11/28): python3-rados-15.0.0-8795.gc6e9f8f.el8 4.7 MB/s | 348 kB 00:00 2019-12-21T15:42:40.374 INFO:teuthology.orchestra.run.smithi172.stdout:(12/28): python3-rbd-15.0.0-8795.gc6e9f8f.el8.x 4.0 MB/s | 302 kB 00:00 2019-12-21T15:42:40.444 INFO:teuthology.orchestra.run.smithi172.stdout:(13/28): python3-rgw-15.0.0-8795.gc6e9f8f.el8.x 1.6 MB/s | 113 kB 00:00 2019-12-21T15:42:40.503 INFO:teuthology.orchestra.run.smithi172.stdout:(14/28): gperftools-libs-2.7-6.el8.x86_64.rpm 5.2 MB/s | 315 kB 00:00 2019-12-21T15:42:40.513 INFO:teuthology.orchestra.run.smithi172.stdout:(15/28): liboath-2.6.2-3.el8.x86_64.rpm 5.8 MB/s | 59 kB 00:00 2019-12-21T15:42:40.523 INFO:teuthology.orchestra.run.smithi172.stdout:(16/28): libunwind-1.3.1-3.el8.x86_64.rpm 7.7 MB/s | 75 kB 00:00 2019-12-21T15:42:40.527 INFO:teuthology.orchestra.run.smithi172.stdout:(17/28): leveldb-1.21-2.el8.x86_64.rpm 39 MB/s | 179 kB 00:00 2019-12-21T15:42:40.530 INFO:teuthology.orchestra.run.smithi172.stdout:(18/28): libbabeltrace-1.5.6-2.el8.x86_64.rpm 102 MB/s | 200 kB 00:00 2019-12-21T15:42:40.532 INFO:teuthology.orchestra.run.smithi172.stdout:(19/28): lttng-ust-2.8.1-11.el8.x86_64.rpm 101 MB/s | 257 kB 00:00 2019-12-21T15:42:40.550 INFO:teuthology.orchestra.run.smithi172.stdout:(20/28): librgw2-15.0.0-8795.gc6e9f8f.el8.x86_6 11 MB/s | 5.6 MB 00:00 2019-12-21T15:42:40.704 INFO:teuthology.orchestra.run.smithi172.stdout:(21/28): ceph-common-15.0.0-8795.gc6e9f8f.el8.x 11 MB/s | 20 MB 00:01 2019-12-21T15:42:40.710 INFO:teuthology.orchestra.run.smithi172.stdout:(22/28): python3-prettytable-0.7.2-14.el8.noarc 248 kB/s | 44 kB 00:00 2019-12-21T15:42:40.728 INFO:teuthology.orchestra.run.smithi172.stdout:(23/28): librdkafka-0.11.4-1.el8.x86_64.rpm 2.0 MB/s | 353 kB 00:00 2019-12-21T15:42:40.745 INFO:teuthology.orchestra.run.smithi172.stdout:(24/28): librdmacm-22-2.el8.x86_64.rpm 2.0 MB/s | 70 kB 00:00 2019-12-21T15:42:40.748 INFO:teuthology.orchestra.run.smithi172.stdout:(25/28): rdma-core-22-2.el8.x86_64.rpm 1.3 MB/s | 53 kB 00:00 2019-12-21T15:42:40.764 INFO:teuthology.orchestra.run.smithi172.stdout:(26/28): userspace-rcu-0.10.1-2.el8.x86_64.rpm 3.1 MB/s | 101 kB 00:00 2019-12-21T15:42:40.782 INFO:teuthology.orchestra.run.smithi172.stdout:(27/28): librabbitmq-0.9.0-1.el8.x86_64.rpm 1.3 MB/s | 47 kB 00:00 2019-12-21T15:42:40.784 INFO:teuthology.orchestra.run.smithi172.stdout:(28/28): libibverbs-22-2.el8.x86_64.rpm 7.4 MB/s | 237 kB 00:00 2019-12-21T15:42:40.786 INFO:teuthology.orchestra.run.smithi172.stdout:-------------------------------------------------------------------------------- 2019-12-21T15:42:40.786 INFO:teuthology.orchestra.run.smithi172.stdout:Total 21 MB/s | 46 MB 00:02 2019-12-21T15:42:40.792 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction check 2019-12-21T15:42:40.837 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction check succeeded. 2019-12-21T15:42:40.837 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction test 2019-12-21T15:42:41.133 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction test succeeded. 2019-12-21T15:42:41.134 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction 2019-12-21T15:42:42.163 INFO:teuthology.orchestra.run.smithi172.stdout: Preparing : 1/1 2019-12-21T15:42:42.359 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : liboath-2.6.2-3.el8.x86_64 1/28 2019-12-21T15:42:42.560 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : librabbitmq-0.9.0-1.el8.x86_64 2/28 2019-12-21T15:42:42.729 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : librdkafka-0.11.4-1.el8.x86_64 3/28 2019-12-21T15:42:42.816 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: librdkafka-0.11.4-1.el8.x86_64 3/28 2019-12-21T15:42:43.102 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : rdma-core-22-2.el8.x86_64 4/28 2019-12-21T15:42:43.338 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : libibverbs-22-2.el8.x86_64 5/28 2019-12-21T15:42:43.387 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: libibverbs-22-2.el8.x86_64 5/28 2019-12-21T15:42:43.563 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : librdmacm-22-2.el8.x86_64 6/28 2019-12-21T15:42:43.651 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: librdmacm-22-2.el8.x86_64 6/28 2019-12-21T15:42:43.815 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-ceph-argparse-2:15.0.0-8795.gc6e9f8f.el8.x 7/28 2019-12-21T15:42:43.996 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : userspace-rcu-0.10.1-2.el8.x86_64 8/28 2019-12-21T15:42:44.103 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: userspace-rcu-0.10.1-2.el8.x86_64 8/28 2019-12-21T15:42:44.279 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 9/28 2019-12-21T15:42:44.669 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 9/28 2019-12-21T15:42:44.854 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : librados2-2:15.0.0-8795.gc6e9f8f.el8.x86_64 10/28 2019-12-21T15:42:44.942 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: librados2-2:15.0.0-8795.gc6e9f8f.el8.x86_64 10/28 2019-12-21T15:42:45.137 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-rados-2:15.0.0-8795.gc6e9f8f.el8.x86_64 11/28 2019-12-21T15:42:45.337 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : libcephfs2-2:15.0.0-8795.gc6e9f8f.el8.x86_64 12/28 2019-12-21T15:42:45.556 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: libcephfs2-2:15.0.0-8795.gc6e9f8f.el8.x86_64 12/28 2019-12-21T15:42:45.737 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : librbd1-2:15.0.0-8795.gc6e9f8f.el8.x86_64 13/28 2019-12-21T15:42:46.307 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: librbd1-2:15.0.0-8795.gc6e9f8f.el8.x86_64 13/28 2019-12-21T15:42:46.471 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : librgw2-2:15.0.0-8795.gc6e9f8f.el8.x86_64 14/28 2019-12-21T15:42:46.525 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: librgw2-2:15.0.0-8795.gc6e9f8f.el8.x86_64 14/28 2019-12-21T15:42:46.733 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-rgw-2:15.0.0-8795.gc6e9f8f.el8.x86_64 15/28 2019-12-21T15:42:46.887 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-rbd-2:15.0.0-8795.gc6e9f8f.el8.x86_64 16/28 2019-12-21T15:42:47.111 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-cephfs-2:15.0.0-8795.gc6e9f8f.el8.x86_64 17/28 2019-12-21T15:42:47.296 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : libradosstriper1-2:15.0.0-8795.gc6e9f8f.el8.x86_64 18/28 2019-12-21T15:42:47.366 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: libradosstriper1-2:15.0.0-8795.gc6e9f8f.el8.x86_64 18/28 2019-12-21T15:42:47.520 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 19/28 2019-12-21T15:42:47.722 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : libbabeltrace-1.5.6-2.el8.x86_64 20/28 2019-12-21T15:42:47.925 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 21/28 2019-12-21T15:42:48.129 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : gperftools-libs-2.7-6.el8.x86_64 22/28 2019-12-21T15:42:48.334 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : leveldb-1.21-2.el8.x86_64 23/28 2019-12-21T15:42:48.487 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-ceph-common-2:15.0.0-8795.gc6e9f8f.el8.x86 24/28 2019-12-21T15:42:51.472 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-common-2:15.0.0-8795.gc6e9f8f.el8.x86_64 25/28 2019-12-21T15:42:51.712 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-common-2:15.0.0-8795.gc6e9f8f.el8.x86_64 25/28 2019-12-21T15:42:52.368 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-common-2:15.0.0-8795.gc6e9f8f.el8.x86_64 25/28 2019-12-21T15:42:52.620 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-base-2:15.0.0-8795.gc6e9f8f.el8.x86_64 26/28 2019-12-21T15:42:52.694 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-base-2:15.0.0-8795.gc6e9f8f.el8.x86_64 26/28 2019-12-21T15:42:52.811 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-selinux-2:15.0.0-8795.gc6e9f8f.el8.x86_64 27/28 2019-12-21T15:43:08.873 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-selinux-2:15.0.0-8795.gc6e9f8f.el8.x86_64 27/28 2019-12-21T15:43:08.873 INFO:teuthology.orchestra.run.smithi172.stdout:skipping the directory /sys 2019-12-21T15:43:08.873 INFO:teuthology.orchestra.run.smithi172.stdout:skipping the directory /proc 2019-12-21T15:43:08.873 INFO:teuthology.orchestra.run.smithi172.stdout:skipping the directory /dev 2019-12-21T15:43:08.873 INFO:teuthology.orchestra.run.smithi172.stdout:skipping the directory /run 2019-12-21T15:43:08.874 INFO:teuthology.orchestra.run.smithi172.stdout:skipping the directory /mnt 2019-12-21T15:43:08.874 INFO:teuthology.orchestra.run.smithi172.stdout:skipping the directory /var/tmp 2019-12-21T15:43:08.874 INFO:teuthology.orchestra.run.smithi172.stdout:skipping the directory /home 2019-12-21T15:43:08.874 INFO:teuthology.orchestra.run.smithi172.stdout:skipping the directory /tmp 2019-12-21T15:43:08.874 INFO:teuthology.orchestra.run.smithi172.stdout:skipping the directory /dev 2019-12-21T15:43:08.874 INFO:teuthology.orchestra.run.smithi172.stdout: 2019-12-21T15:43:09.658 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-radosgw-2:15.0.0-8795.gc6e9f8f.el8.x86_64 28/28 2019-12-21T15:43:10.026 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-radosgw-2:15.0.0-8795.gc6e9f8f.el8.x86_64 28/28 2019-12-21T15:43:10.026 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-base-2:15.0.0-8795.gc6e9f8f.el8.x86_64 1/28 2019-12-21T15:43:10.026 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-common-2:15.0.0-8795.gc6e9f8f.el8.x86_64 2/28 2019-12-21T15:43:10.026 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-radosgw-2:15.0.0-8795.gc6e9f8f.el8.x86_64 3/28 2019-12-21T15:43:10.027 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-selinux-2:15.0.0-8795.gc6e9f8f.el8.x86_64 4/28 2019-12-21T15:43:10.027 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : libcephfs2-2:15.0.0-8795.gc6e9f8f.el8.x86_64 5/28 2019-12-21T15:43:10.027 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : librados2-2:15.0.0-8795.gc6e9f8f.el8.x86_64 6/28 2019-12-21T15:43:10.027 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : libradosstriper1-2:15.0.0-8795.gc6e9f8f.el8.x86_64 7/28 2019-12-21T15:43:10.027 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : librbd1-2:15.0.0-8795.gc6e9f8f.el8.x86_64 8/28 2019-12-21T15:43:10.027 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : librgw2-2:15.0.0-8795.gc6e9f8f.el8.x86_64 9/28 2019-12-21T15:43:10.028 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-ceph-argparse-2:15.0.0-8795.gc6e9f8f.el8.x 10/28 2019-12-21T15:43:10.028 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-ceph-common-2:15.0.0-8795.gc6e9f8f.el8.x86 11/28 2019-12-21T15:43:10.028 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-cephfs-2:15.0.0-8795.gc6e9f8f.el8.x86_64 12/28 2019-12-21T15:43:10.028 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-rados-2:15.0.0-8795.gc6e9f8f.el8.x86_64 13/28 2019-12-21T15:43:10.028 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-rbd-2:15.0.0-8795.gc6e9f8f.el8.x86_64 14/28 2019-12-21T15:43:10.028 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-rgw-2:15.0.0-8795.gc6e9f8f.el8.x86_64 15/28 2019-12-21T15:43:10.029 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : gperftools-libs-2.7-6.el8.x86_64 16/28 2019-12-21T15:43:10.029 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 17/28 2019-12-21T15:43:10.029 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 18/28 2019-12-21T15:43:10.029 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : leveldb-1.21-2.el8.x86_64 19/28 2019-12-21T15:43:10.029 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : libbabeltrace-1.5.6-2.el8.x86_64 20/28 2019-12-21T15:43:10.029 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 21/28 2019-12-21T15:43:10.030 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 22/28 2019-12-21T15:43:10.030 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : librdkafka-0.11.4-1.el8.x86_64 23/28 2019-12-21T15:43:10.030 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : rdma-core-22-2.el8.x86_64 24/28 2019-12-21T15:43:10.030 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : librdmacm-22-2.el8.x86_64 25/28 2019-12-21T15:43:10.030 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : userspace-rcu-0.10.1-2.el8.x86_64 26/28 2019-12-21T15:43:10.031 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : librabbitmq-0.9.0-1.el8.x86_64 27/28 2019-12-21T15:43:10.849 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : libibverbs-22-2.el8.x86_64 28/28 2019-12-21T15:43:10.849 INFO:teuthology.orchestra.run.smithi172.stdout:Installed products updated. 2019-12-21T15:43:12.277 INFO:teuthology.orchestra.run.smithi172.stdout: 2019-12-21T15:43:12.277 INFO:teuthology.orchestra.run.smithi172.stdout:Installed: 2019-12-21T15:43:12.278 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-radosgw-2:15.0.0-8795.gc6e9f8f.el8.x86_64 2019-12-21T15:43:12.278 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-base-2:15.0.0-8795.gc6e9f8f.el8.x86_64 2019-12-21T15:43:12.278 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-common-2:15.0.0-8795.gc6e9f8f.el8.x86_64 2019-12-21T15:43:12.278 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-selinux-2:15.0.0-8795.gc6e9f8f.el8.x86_64 2019-12-21T15:43:12.278 INFO:teuthology.orchestra.run.smithi172.stdout: libcephfs2-2:15.0.0-8795.gc6e9f8f.el8.x86_64 2019-12-21T15:43:12.278 INFO:teuthology.orchestra.run.smithi172.stdout: librados2-2:15.0.0-8795.gc6e9f8f.el8.x86_64 2019-12-21T15:43:12.279 INFO:teuthology.orchestra.run.smithi172.stdout: libradosstriper1-2:15.0.0-8795.gc6e9f8f.el8.x86_64 2019-12-21T15:43:12.279 INFO:teuthology.orchestra.run.smithi172.stdout: librbd1-2:15.0.0-8795.gc6e9f8f.el8.x86_64 2019-12-21T15:43:12.279 INFO:teuthology.orchestra.run.smithi172.stdout: librgw2-2:15.0.0-8795.gc6e9f8f.el8.x86_64 2019-12-21T15:43:12.279 INFO:teuthology.orchestra.run.smithi172.stdout: python3-ceph-argparse-2:15.0.0-8795.gc6e9f8f.el8.x86_64 2019-12-21T15:43:12.279 INFO:teuthology.orchestra.run.smithi172.stdout: python3-ceph-common-2:15.0.0-8795.gc6e9f8f.el8.x86_64 2019-12-21T15:43:12.279 INFO:teuthology.orchestra.run.smithi172.stdout: python3-cephfs-2:15.0.0-8795.gc6e9f8f.el8.x86_64 2019-12-21T15:43:12.279 INFO:teuthology.orchestra.run.smithi172.stdout: python3-rados-2:15.0.0-8795.gc6e9f8f.el8.x86_64 2019-12-21T15:43:12.280 INFO:teuthology.orchestra.run.smithi172.stdout: python3-rbd-2:15.0.0-8795.gc6e9f8f.el8.x86_64 2019-12-21T15:43:12.280 INFO:teuthology.orchestra.run.smithi172.stdout: python3-rgw-2:15.0.0-8795.gc6e9f8f.el8.x86_64 2019-12-21T15:43:12.280 INFO:teuthology.orchestra.run.smithi172.stdout: gperftools-libs-2.7-6.el8.x86_64 2019-12-21T15:43:12.280 INFO:teuthology.orchestra.run.smithi172.stdout: liboath-2.6.2-3.el8.x86_64 2019-12-21T15:43:12.280 INFO:teuthology.orchestra.run.smithi172.stdout: libunwind-1.3.1-3.el8.x86_64 2019-12-21T15:43:12.280 INFO:teuthology.orchestra.run.smithi172.stdout: leveldb-1.21-2.el8.x86_64 2019-12-21T15:43:12.280 INFO:teuthology.orchestra.run.smithi172.stdout: libbabeltrace-1.5.6-2.el8.x86_64 2019-12-21T15:43:12.281 INFO:teuthology.orchestra.run.smithi172.stdout: lttng-ust-2.8.1-11.el8.x86_64 2019-12-21T15:43:12.281 INFO:teuthology.orchestra.run.smithi172.stdout: python3-prettytable-0.7.2-14.el8.noarch 2019-12-21T15:43:12.281 INFO:teuthology.orchestra.run.smithi172.stdout: librdkafka-0.11.4-1.el8.x86_64 2019-12-21T15:43:12.281 INFO:teuthology.orchestra.run.smithi172.stdout: rdma-core-22-2.el8.x86_64 2019-12-21T15:43:12.281 INFO:teuthology.orchestra.run.smithi172.stdout: librdmacm-22-2.el8.x86_64 2019-12-21T15:43:12.281 INFO:teuthology.orchestra.run.smithi172.stdout: userspace-rcu-0.10.1-2.el8.x86_64 2019-12-21T15:43:12.281 INFO:teuthology.orchestra.run.smithi172.stdout: librabbitmq-0.9.0-1.el8.x86_64 2019-12-21T15:43:12.282 INFO:teuthology.orchestra.run.smithi172.stdout: libibverbs-22-2.el8.x86_64 2019-12-21T15:43:12.282 INFO:teuthology.orchestra.run.smithi172.stdout: 2019-12-21T15:43:12.282 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2019-12-21T15:43:12.461 INFO:teuthology.orchestra.run.smithi172:> sudo yum -y install ceph-test 2019-12-21T15:43:12.801 INFO:teuthology.orchestra.run.smithi172.stdout:Updating Subscription Management repositories. 2019-12-21T15:43:13.310 INFO:teuthology.orchestra.run.smithi172.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 88 kB/s | 2.8 kB 00:00 2019-12-21T15:43:13.371 INFO:teuthology.orchestra.run.smithi172.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 78 kB/s | 2.3 kB 00:00 2019-12-21T15:43:13.893 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2019-12-21T15:43:13.894 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2019-12-21T15:43:13.894 INFO:teuthology.orchestra.run.smithi172.stdout: Package Arch Version Repository Size 2019-12-21T15:43:13.895 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2019-12-21T15:43:13.895 INFO:teuthology.orchestra.run.smithi172.stdout:Installing: 2019-12-21T15:43:13.895 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-test x86_64 2:15.0.0-8795.gc6e9f8f.el8 2019-12-21T15:43:13.895 INFO:teuthology.orchestra.run.smithi172.stdout: ceph 34 M 2019-12-21T15:43:13.895 INFO:teuthology.orchestra.run.smithi172.stdout:Installing dependencies: 2019-12-21T15:43:13.895 INFO:teuthology.orchestra.run.smithi172.stdout: xmlstarlet x86_64 1.6.1-13.el8 lab-extras 69 k 2019-12-21T15:43:13.895 INFO:teuthology.orchestra.run.smithi172.stdout: jq x86_64 1.5-12.el8 rhel-8-for-x86_64-appstream-rpms 161 k 2019-12-21T15:43:13.896 INFO:teuthology.orchestra.run.smithi172.stdout: socat x86_64 1.7.3.2-6.el8 rhel-8-for-x86_64-appstream-rpms 298 k 2019-12-21T15:43:13.896 INFO:teuthology.orchestra.run.smithi172.stdout: oniguruma x86_64 6.8.2-1.el8 rhel-8-for-x86_64-appstream-rpms 188 k 2019-12-21T15:43:13.896 INFO:teuthology.orchestra.run.smithi172.stdout: libxslt x86_64 1.1.32-3.el8 rhel-8-for-x86_64-baseos-rpms 249 k 2019-12-21T15:43:13.896 INFO:teuthology.orchestra.run.smithi172.stdout: 2019-12-21T15:43:13.896 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction Summary 2019-12-21T15:43:13.896 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2019-12-21T15:43:13.897 INFO:teuthology.orchestra.run.smithi172.stdout:Install 6 Packages 2019-12-21T15:43:13.897 INFO:teuthology.orchestra.run.smithi172.stdout: 2019-12-21T15:43:13.897 INFO:teuthology.orchestra.run.smithi172.stdout:Total download size: 35 M 2019-12-21T15:43:13.897 INFO:teuthology.orchestra.run.smithi172.stdout:Installed size: 150 M 2019-12-21T15:43:13.898 INFO:teuthology.orchestra.run.smithi172.stdout:Downloading Packages: 2019-12-21T15:43:13.900 INFO:teuthology.orchestra.run.smithi172.stdout:(1/6): xmlstarlet-1.6.1-13.el8.x86_64.rpm 16 MB/s | 69 kB 00:00 2019-12-21T15:43:13.938 INFO:teuthology.orchestra.run.smithi172.stdout:(2/6): jq-1.5-12.el8.x86_64.rpm 3.8 MB/s | 161 kB 00:00 2019-12-21T15:43:13.952 INFO:teuthology.orchestra.run.smithi172.stdout:(3/6): socat-1.7.3.2-6.el8.x86_64.rpm 6.3 MB/s | 298 kB 00:00 2019-12-21T15:43:13.966 INFO:teuthology.orchestra.run.smithi172.stdout:(4/6): oniguruma-6.8.2-1.el8.x86_64.rpm 8.1 MB/s | 188 kB 00:00 2019-12-21T15:43:13.992 INFO:teuthology.orchestra.run.smithi172.stdout:(5/6): libxslt-1.1.32-3.el8.x86_64.rpm 6.9 MB/s | 249 kB 00:00 2019-12-21T15:43:15.512 INFO:teuthology.orchestra.run.smithi172.stdout:(6/6): ceph-test-15.0.0-8795.gc6e9f8f.el8.x86_6 21 MB/s | 34 MB 00:01 2019-12-21T15:43:15.514 INFO:teuthology.orchestra.run.smithi172.stdout:-------------------------------------------------------------------------------- 2019-12-21T15:43:15.514 INFO:teuthology.orchestra.run.smithi172.stdout:Total 21 MB/s | 35 MB 00:01 2019-12-21T15:43:15.518 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction check 2019-12-21T15:43:15.538 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction check succeeded. 2019-12-21T15:43:15.538 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction test 2019-12-21T15:43:15.784 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction test succeeded. 2019-12-21T15:43:15.785 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction 2019-12-21T15:43:16.171 INFO:teuthology.orchestra.run.smithi172.stdout: Preparing : 1/1 2019-12-21T15:43:16.366 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : libxslt-1.1.32-3.el8.x86_64 1/6 2019-12-21T15:43:16.535 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : xmlstarlet-1.6.1-13.el8.x86_64 2/6 2019-12-21T15:43:16.707 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : oniguruma-6.8.2-1.el8.x86_64 3/6 2019-12-21T15:43:16.795 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: oniguruma-6.8.2-1.el8.x86_64 3/6 2019-12-21T15:43:17.129 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : jq-1.5-12.el8.x86_64 4/6 2019-12-21T15:43:20.972 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : socat-1.7.3.2-6.el8.x86_64 5/6 2019-12-21T15:43:21.220 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-test-2:15.0.0-8795.gc6e9f8f.el8.x86_64 6/6 2019-12-21T15:43:21.406 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-test-2:15.0.0-8795.gc6e9f8f.el8.x86_64 6/6 2019-12-21T15:43:21.406 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-test-2:15.0.0-8795.gc6e9f8f.el8.x86_64 1/6 2019-12-21T15:43:21.406 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : xmlstarlet-1.6.1-13.el8.x86_64 2/6 2019-12-21T15:43:21.406 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : jq-1.5-12.el8.x86_64 3/6 2019-12-21T15:43:21.407 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : socat-1.7.3.2-6.el8.x86_64 4/6 2019-12-21T15:43:21.407 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : oniguruma-6.8.2-1.el8.x86_64 5/6 2019-12-21T15:43:21.790 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : libxslt-1.1.32-3.el8.x86_64 6/6 2019-12-21T15:43:21.790 INFO:teuthology.orchestra.run.smithi172.stdout:Installed products updated. 2019-12-21T15:43:23.417 INFO:teuthology.orchestra.run.smithi172.stdout: 2019-12-21T15:43:23.417 INFO:teuthology.orchestra.run.smithi172.stdout:Installed: 2019-12-21T15:43:23.417 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-test-2:15.0.0-8795.gc6e9f8f.el8.x86_64 xmlstarlet-1.6.1-13.el8.x86_64 2019-12-21T15:43:23.418 INFO:teuthology.orchestra.run.smithi172.stdout: jq-1.5-12.el8.x86_64 socat-1.7.3.2-6.el8.x86_64 2019-12-21T15:43:23.418 INFO:teuthology.orchestra.run.smithi172.stdout: oniguruma-6.8.2-1.el8.x86_64 libxslt-1.1.32-3.el8.x86_64 2019-12-21T15:43:23.418 INFO:teuthology.orchestra.run.smithi172.stdout: 2019-12-21T15:43:23.418 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2019-12-21T15:43:23.872 INFO:teuthology.orchestra.run.smithi172:> sudo yum -y install ceph 2019-12-21T15:43:24.199 INFO:teuthology.orchestra.run.smithi172.stdout:Updating Subscription Management repositories. 2019-12-21T15:43:24.786 INFO:teuthology.orchestra.run.smithi172.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 84 kB/s | 2.8 kB 00:00 2019-12-21T15:43:24.854 INFO:teuthology.orchestra.run.smithi172.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 61 kB/s | 2.3 kB 00:00 2019-12-21T15:43:25.379 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2019-12-21T15:43:25.383 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2019-12-21T15:43:25.383 INFO:teuthology.orchestra.run.smithi172.stdout: Package ArchVersion Repository Size 2019-12-21T15:43:25.385 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2019-12-21T15:43:25.385 INFO:teuthology.orchestra.run.smithi172.stdout:Installing: 2019-12-21T15:43:25.385 INFO:teuthology.orchestra.run.smithi172.stdout: ceph x86_642:15.0.0-8795.gc6e9f8f.el8 ceph 6.4 k 2019-12-21T15:43:25.385 INFO:teuthology.orchestra.run.smithi172.stdout:Installing dependencies: 2019-12-21T15:43:25.385 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mds x86_642:15.0.0-8795.gc6e9f8f.el8 ceph 2.0 M 2019-12-21T15:43:25.385 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mgr x86_642:15.0.0-8795.gc6e9f8f.el8 ceph 1.5 M 2019-12-21T15:43:25.386 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mon x86_642:15.0.0-8795.gc6e9f8f.el8 ceph 4.3 M 2019-12-21T15:43:25.386 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-osd x86_642:15.0.0-8795.gc6e9f8f.el8 ceph 17 M 2019-12-21T15:43:25.386 INFO:teuthology.orchestra.run.smithi172.stdout: python3-cheroot noarch8.2.1-1.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 145 k 2019-12-21T15:43:25.386 INFO:teuthology.orchestra.run.smithi172.stdout: python3-cherrypy noarch18.4.0-1.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 385 k 2019-12-21T15:43:25.387 INFO:teuthology.orchestra.run.smithi172.stdout: python3-jaraco noarch6.2-6.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 11 k 2019-12-21T15:43:25.387 INFO:teuthology.orchestra.run.smithi172.stdout: python3-jaraco-functools noarch2.0-4.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 18 k 2019-12-21T15:43:25.387 INFO:teuthology.orchestra.run.smithi172.stdout: python3-logutils noarch0.3.5-11.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 49 k 2019-12-21T15:43:25.387 INFO:teuthology.orchestra.run.smithi172.stdout: python3-more-itertools noarch7.2.0-3.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 59 k 2019-12-21T15:43:25.387 INFO:teuthology.orchestra.run.smithi172.stdout: python3-pecan noarch1.3.2-9.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 283 k 2019-12-21T15:43:25.388 INFO:teuthology.orchestra.run.smithi172.stdout: python3-portend noarch2.6-1.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 16 k 2019-12-21T15:43:25.388 INFO:teuthology.orchestra.run.smithi172.stdout: python3-simplegeneric noarch0.8.1-17.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 19 k 2019-12-21T15:43:25.388 INFO:teuthology.orchestra.run.smithi172.stdout: python3-singledispatch noarch3.4.0.3-18.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 24 k 2019-12-21T15:43:25.388 INFO:teuthology.orchestra.run.smithi172.stdout: python3-tempora noarch1.14.1-5.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 30 k 2019-12-21T15:43:25.388 INFO:teuthology.orchestra.run.smithi172.stdout: python3-trustme noarch0.5.2-4.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 27 k 2019-12-21T15:43:25.388 INFO:teuthology.orchestra.run.smithi172.stdout: python3-zc-lockfile noarch2.0-2.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 23 k 2019-12-21T15:43:25.389 INFO:teuthology.orchestra.run.smithi172.stdout: python3-bcrypt x86_643.1.6-2.el8.1 epel 44 k 2019-12-21T15:43:25.389 INFO:teuthology.orchestra.run.smithi172.stdout: python3-beautifulsoup4 noarch4.6.3-2.el8.1 epel 185 k 2019-12-21T15:43:25.389 INFO:teuthology.orchestra.run.smithi172.stdout: python3-waitress noarch1.2.1-2.el8.1 epel 241 k 2019-12-21T15:43:25.389 INFO:teuthology.orchestra.run.smithi172.stdout: python3-webob noarch1.8.5-1.el8.1 epel 251 k 2019-12-21T15:43:25.389 INFO:teuthology.orchestra.run.smithi172.stdout: python3-webtest noarch2.0.33-1.el8 epel 85 k 2019-12-21T15:43:25.389 INFO:teuthology.orchestra.run.smithi172.stdout: python3-webencodings noarch0.5.1-6.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2019-12-21T15:43:25.390 INFO:teuthology.orchestra.run.smithi172.stdout: python3-lxml x86_644.2.3-1.el8 rhel-8-for-x86_64-appstream-rpms 1.5 M 2019-12-21T15:43:25.390 INFO:teuthology.orchestra.run.smithi172.stdout: python3-werkzeug noarch0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2019-12-21T15:43:25.390 INFO:teuthology.orchestra.run.smithi172.stdout: python3-mako noarch1.0.6-13.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2019-12-21T15:43:25.390 INFO:teuthology.orchestra.run.smithi172.stdout: python3-pytz noarch2017.2-9.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2019-12-21T15:43:25.390 INFO:teuthology.orchestra.run.smithi172.stdout: python3-markupsafe x86_640.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2019-12-21T15:43:25.390 INFO:teuthology.orchestra.run.smithi172.stdout:Installing weak dependencies: 2019-12-21T15:43:25.391 INFO:teuthology.orchestra.run.smithi172.stdout: python3-cssselect noarch0.9.2-10.el8 epel 40 k 2019-12-21T15:43:25.391 INFO:teuthology.orchestra.run.smithi172.stdout: python3-html5lib noarch1:0.999999999-6.el8 rhel-8-for-x86_64-appstream-rpms 214 k 2019-12-21T15:43:25.391 INFO:teuthology.orchestra.run.smithi172.stdout: 2019-12-21T15:43:25.391 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction Summary 2019-12-21T15:43:25.391 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2019-12-21T15:43:25.391 INFO:teuthology.orchestra.run.smithi172.stdout:Install 31 Packages 2019-12-21T15:43:25.392 INFO:teuthology.orchestra.run.smithi172.stdout: 2019-12-21T15:43:25.392 INFO:teuthology.orchestra.run.smithi172.stdout:Total download size: 29 M 2019-12-21T15:43:25.392 INFO:teuthology.orchestra.run.smithi172.stdout:Installed size: 102 M 2019-12-21T15:43:25.393 INFO:teuthology.orchestra.run.smithi172.stdout:Downloading Packages: 2019-12-21T15:43:26.639 INFO:teuthology.orchestra.run.smithi172.stdout:(1/31): ceph-15.0.0-8795.gc6e9f8f.el8.x86_64.rp 67 kB/s | 6.4 kB 00:00 2019-12-21T15:43:26.851 INFO:teuthology.orchestra.run.smithi172.stdout:(2/31): ceph-mds-15.0.0-8795.gc6e9f8f.el8.x86_6 6.6 MB/s | 2.0 MB 00:00 2019-12-21T15:43:26.869 INFO:teuthology.orchestra.run.smithi172.stdout:(3/31): ceph-mgr-15.0.0-8795.gc6e9f8f.el8.x86_6 4.7 MB/s | 1.5 MB 00:00 2019-12-21T15:43:27.167 INFO:teuthology.orchestra.run.smithi172.stdout:(4/31): ceph-mon-15.0.0-8795.gc6e9f8f.el8.x86_6 8.3 MB/s | 4.3 MB 00:00 2019-12-21T15:43:27.331 INFO:teuthology.orchestra.run.smithi172.stdout:(5/31): python3-cheroot-8.2.1-1.el8.noarch.rpm 314 kB/s | 145 kB 00:00 2019-12-21T15:43:27.409 INFO:teuthology.orchestra.run.smithi172.stdout:(6/31): python3-jaraco-6.2-6.el8.noarch.rpm 149 kB/s | 11 kB 00:00 2019-12-21T15:43:27.492 INFO:teuthology.orchestra.run.smithi172.stdout:(7/31): python3-jaraco-functools-2.0-4.el8.noar 216 kB/s | 18 kB 00:00 2019-12-21T15:43:27.643 INFO:teuthology.orchestra.run.smithi172.stdout:(8/31): ceph-osd-15.0.0-8795.gc6e9f8f.el8.x86_6 21 MB/s | 17 MB 00:00 2019-12-21T15:43:27.644 INFO:teuthology.orchestra.run.smithi172.stdout:(9/31): python3-logutils-0.3.5-11.el8.noarch.rp 322 kB/s | 49 kB 00:00 2019-12-21T15:43:27.788 INFO:teuthology.orchestra.run.smithi172.stdout:(10/31): python3-cherrypy-18.4.0-1.el8.noarch.r 620 kB/s | 385 kB 00:00 2019-12-21T15:43:27.844 INFO:teuthology.orchestra.run.smithi172.stdout:(11/31): python3-pecan-1.3.2-9.el8.noarch.rpm 1.4 MB/s | 283 kB 00:00 2019-12-21T15:43:27.912 INFO:teuthology.orchestra.run.smithi172.stdout:(12/31): python3-portend-2.6-1.el8.noarch.rpm 130 kB/s | 16 kB 00:00 2019-12-21T15:43:27.962 INFO:teuthology.orchestra.run.smithi172.stdout:(13/31): python3-simplegeneric-0.8.1-17.el8.noa 163 kB/s | 19 kB 00:00 2019-12-21T15:43:27.992 INFO:teuthology.orchestra.run.smithi172.stdout:(14/31): python3-more-itertools-7.2.0-3.el8.noa 170 kB/s | 59 kB 00:00 2019-12-21T15:43:28.038 INFO:teuthology.orchestra.run.smithi172.stdout:(15/31): python3-singledispatch-3.4.0.3-18.el8. 192 kB/s | 24 kB 00:00 2019-12-21T15:43:28.063 INFO:teuthology.orchestra.run.smithi172.stdout:(16/31): python3-tempora-1.14.1-5.el8.noarch.rp 296 kB/s | 30 kB 00:00 2019-12-21T15:43:28.063 INFO:teuthology.orchestra.run.smithi172.stdout:(17/31): python3-trustme-0.5.2-4.el8.noarch.rpm 377 kB/s | 27 kB 00:00 2019-12-21T15:43:28.135 INFO:teuthology.orchestra.run.smithi172.stdout:(18/31): python3-zc-lockfile-2.0-2.el8.noarch.r 243 kB/s | 23 kB 00:00 2019-12-21T15:43:28.241 INFO:teuthology.orchestra.run.smithi172.stdout:(19/31): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 245 kB/s | 44 kB 00:00 2019-12-21T15:43:28.275 INFO:teuthology.orchestra.run.smithi172.stdout:(20/31): python3-beautifulsoup4-4.6.3-2.el8.1.n 879 kB/s | 185 kB 00:00 2019-12-21T15:43:28.308 INFO:teuthology.orchestra.run.smithi172.stdout:(21/31): python3-cssselect-0.9.2-10.el8.noarch. 233 kB/s | 40 kB 00:00 2019-12-21T15:43:28.341 INFO:teuthology.orchestra.run.smithi172.stdout:(22/31): python3-waitress-1.2.1-2.el8.1.noarch. 2.4 MB/s | 241 kB 00:00 2019-12-21T15:43:28.342 INFO:teuthology.orchestra.run.smithi172.stdout:(23/31): python3-webob-1.8.5-1.el8.1.noarch.rpm 3.6 MB/s | 251 kB 00:00 2019-12-21T15:43:28.379 INFO:teuthology.orchestra.run.smithi172.stdout:(24/31): python3-webtest-2.0.33-1.el8.noarch.rp 1.2 MB/s | 85 kB 00:00 2019-12-21T15:43:28.385 INFO:teuthology.orchestra.run.smithi172.stdout:(25/31): python3-webencodings-0.5.1-6.el8.noarc 606 kB/s | 27 kB 00:00 2019-12-21T15:43:28.425 INFO:teuthology.orchestra.run.smithi172.stdout:(26/31): python3-lxml-4.2.3-1.el8.x86_64.rpm 20 MB/s | 1.5 MB 00:00 2019-12-21T15:43:28.426 INFO:teuthology.orchestra.run.smithi172.stdout:(27/31): python3-mako-1.0.6-13.el8.noarch.rpm 4.3 MB/s | 157 kB 00:00 2019-12-21T15:43:28.434 INFO:teuthology.orchestra.run.smithi172.stdout:(28/31): python3-werkzeug-0.12.2-4.el8.noarch.r 8.1 MB/s | 457 kB 00:00 2019-12-21T15:43:28.451 INFO:teuthology.orchestra.run.smithi172.stdout:(29/31): python3-html5lib-0.999999999-6.el8.noa 6.6 MB/s | 214 kB 00:00 2019-12-21T15:43:28.462 INFO:teuthology.orchestra.run.smithi172.stdout:(30/31): python3-pytz-2017.2-9.el8.noarch.rpm 1.3 MB/s | 54 kB 00:00 2019-12-21T15:43:28.464 INFO:teuthology.orchestra.run.smithi172.stdout:(31/31): python3-markupsafe-0.23-19.el8.x86_64. 1.5 MB/s | 39 kB 00:00 2019-12-21T15:43:28.469 INFO:teuthology.orchestra.run.smithi172.stdout:-------------------------------------------------------------------------------- 2019-12-21T15:43:28.469 INFO:teuthology.orchestra.run.smithi172.stdout:Total 9.3 MB/s | 29 MB 00:03 2019-12-21T15:43:28.470 INFO:teuthology.orchestra.run.smithi172.stderr:warning: /var/cache/dnf/copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8-ac801414381f5e61/packages/python3-cheroot-8.2.1-1.el8.noarch.rpm: Header V3 RSA/SHA1 Signature, key ID 2a8a41ec: NOKEY 2019-12-21T15:43:28.715 INFO:teuthology.orchestra.run.smithi172.stdout:Copr repo for ceph-el8 owned by ktdreyer 4.0 kB/s | 1.0 kB 00:00 2019-12-21T15:43:28.818 INFO:teuthology.orchestra.run.smithi172.stderr:Importing GPG key 0x2A8A41EC: 2019-12-21T15:43:28.818 INFO:teuthology.orchestra.run.smithi172.stderr: Userid : "ktdreyer_ceph-el8 (None) " 2019-12-21T15:43:28.818 INFO:teuthology.orchestra.run.smithi172.stderr: Fingerprint: 64D3 346F FD63 5D2D B5D0 5873 1014 BDBE 2A8A 41EC 2019-12-21T15:43:28.818 INFO:teuthology.orchestra.run.smithi172.stderr: From : https://copr-be.cloud.fedoraproject.org/results/ktdreyer/ceph-el8/pubkey.gpg 2019-12-21T15:43:28.914 INFO:teuthology.orchestra.run.smithi172.stdout:Key imported successfully 2019-12-21T15:43:28.932 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction check 2019-12-21T15:43:28.969 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction check succeeded. 2019-12-21T15:43:28.969 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction test 2019-12-21T15:43:29.239 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction test succeeded. 2019-12-21T15:43:29.240 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction 2019-12-21T15:43:30.378 INFO:teuthology.orchestra.run.smithi172.stdout: Preparing : 1/1 2019-12-21T15:43:30.583 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 1/31 2019-12-21T15:43:30.766 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 2/31 2019-12-21T15:43:31.025 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 3/31 2019-12-21T15:43:31.258 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 4/31 2019-12-21T15:43:31.514 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 5/31 2019-12-21T15:43:31.749 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 6/31 2019-12-21T15:43:31.982 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-webencodings-0.5.1-6.el8.noarch 7/31 2019-12-21T15:43:32.226 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-html5lib-1:0.999999999-6.el8.noarch 8/31 2019-12-21T15:43:32.423 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-waitress-1.2.1-2.el8.1.noarch 9/31 2019-12-21T15:43:32.745 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-cssselect-0.9.2-10.el8.noarch 10/31 2019-12-21T15:43:33.056 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-lxml-4.2.3-1.el8.x86_64 11/31 2019-12-21T15:43:33.271 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 12/31 2019-12-21T15:43:33.497 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 13/31 2019-12-21T15:43:33.737 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 14/31 2019-12-21T15:43:33.912 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 15/31 2019-12-21T15:43:34.044 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-trustme-0.5.2-4.el8.noarch 16/31 2019-12-21T15:43:34.192 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 17/31 2019-12-21T15:43:34.365 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 18/31 2019-12-21T15:43:34.679 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 19/31 2019-12-21T15:43:34.898 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 20/31 2019-12-21T15:43:35.043 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 21/31 2019-12-21T15:43:35.249 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 22/31 2019-12-21T15:43:35.431 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-cheroot-8.2.1-1.el8.noarch 23/31 2019-12-21T15:43:35.634 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 24/31 2019-12-21T15:43:35.870 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-portend-2.6-1.el8.noarch 25/31 2019-12-21T15:43:36.266 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 26/31 2019-12-21T15:43:36.445 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-mgr-2:15.0.0-8795.gc6e9f8f.el8.x86_64 27/31 2019-12-21T15:43:37.843 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-mgr-2:15.0.0-8795.gc6e9f8f.el8.x86_64 27/31 2019-12-21T15:43:38.020 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-osd-2:15.0.0-8795.gc6e9f8f.el8.x86_64 28/31 2019-12-21T15:43:38.409 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-osd-2:15.0.0-8795.gc6e9f8f.el8.x86_64 28/31 2019-12-21T15:43:38.562 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-mon-2:15.0.0-8795.gc6e9f8f.el8.x86_64 29/31 2019-12-21T15:43:38.750 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-mon-2:15.0.0-8795.gc6e9f8f.el8.x86_64 29/31 2019-12-21T15:43:38.928 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-mds-2:15.0.0-8795.gc6e9f8f.el8.x86_64 30/31 2019-12-21T15:43:38.944 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-mds-2:15.0.0-8795.gc6e9f8f.el8.x86_64 30/31 2019-12-21T15:43:39.194 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-2:15.0.0-8795.gc6e9f8f.el8.x86_64 31/31 2019-12-21T15:43:39.516 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-2:15.0.0-8795.gc6e9f8f.el8.x86_64 31/31 2019-12-21T15:43:39.516 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-2:15.0.0-8795.gc6e9f8f.el8.x86_64 1/31 2019-12-21T15:43:39.516 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-mds-2:15.0.0-8795.gc6e9f8f.el8.x86_64 2/31 2019-12-21T15:43:39.516 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-mgr-2:15.0.0-8795.gc6e9f8f.el8.x86_64 3/31 2019-12-21T15:43:39.516 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-mon-2:15.0.0-8795.gc6e9f8f.el8.x86_64 4/31 2019-12-21T15:43:39.516 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-osd-2:15.0.0-8795.gc6e9f8f.el8.x86_64 5/31 2019-12-21T15:43:39.517 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-cheroot-8.2.1-1.el8.noarch 6/31 2019-12-21T15:43:39.517 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 7/31 2019-12-21T15:43:39.517 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 8/31 2019-12-21T15:43:39.517 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 9/31 2019-12-21T15:43:39.517 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 10/31 2019-12-21T15:43:39.517 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 11/31 2019-12-21T15:43:39.517 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 12/31 2019-12-21T15:43:39.518 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-portend-2.6-1.el8.noarch 13/31 2019-12-21T15:43:39.518 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 14/31 2019-12-21T15:43:39.518 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 15/31 2019-12-21T15:43:39.518 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 16/31 2019-12-21T15:43:39.518 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-trustme-0.5.2-4.el8.noarch 17/31 2019-12-21T15:43:39.518 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 18/31 2019-12-21T15:43:39.518 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 19/31 2019-12-21T15:43:39.519 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 20/31 2019-12-21T15:43:39.519 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-cssselect-0.9.2-10.el8.noarch 21/31 2019-12-21T15:43:39.519 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-waitress-1.2.1-2.el8.1.noarch 22/31 2019-12-21T15:43:39.519 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 23/31 2019-12-21T15:43:39.519 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 24/31 2019-12-21T15:43:39.519 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-webencodings-0.5.1-6.el8.noarch 25/31 2019-12-21T15:43:39.519 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-lxml-4.2.3-1.el8.x86_64 26/31 2019-12-21T15:43:39.520 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 27/31 2019-12-21T15:43:39.520 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 28/31 2019-12-21T15:43:39.520 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-html5lib-1:0.999999999-6.el8.noarch 29/31 2019-12-21T15:43:39.521 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 30/31 2019-12-21T15:43:40.298 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 31/31 2019-12-21T15:43:40.298 INFO:teuthology.orchestra.run.smithi172.stdout:Installed products updated. 2019-12-21T15:43:42.023 INFO:teuthology.orchestra.run.smithi172.stdout: 2019-12-21T15:43:42.023 INFO:teuthology.orchestra.run.smithi172.stdout:Installed: 2019-12-21T15:43:42.023 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-2:15.0.0-8795.gc6e9f8f.el8.x86_64 2019-12-21T15:43:42.023 INFO:teuthology.orchestra.run.smithi172.stdout: python3-cssselect-0.9.2-10.el8.noarch 2019-12-21T15:43:42.024 INFO:teuthology.orchestra.run.smithi172.stdout: python3-html5lib-1:0.999999999-6.el8.noarch 2019-12-21T15:43:42.024 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mds-2:15.0.0-8795.gc6e9f8f.el8.x86_64 2019-12-21T15:43:42.024 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mgr-2:15.0.0-8795.gc6e9f8f.el8.x86_64 2019-12-21T15:43:42.024 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mon-2:15.0.0-8795.gc6e9f8f.el8.x86_64 2019-12-21T15:43:42.024 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-osd-2:15.0.0-8795.gc6e9f8f.el8.x86_64 2019-12-21T15:43:42.024 INFO:teuthology.orchestra.run.smithi172.stdout: python3-cheroot-8.2.1-1.el8.noarch 2019-12-21T15:43:42.024 INFO:teuthology.orchestra.run.smithi172.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2019-12-21T15:43:42.025 INFO:teuthology.orchestra.run.smithi172.stdout: python3-jaraco-6.2-6.el8.noarch 2019-12-21T15:43:42.025 INFO:teuthology.orchestra.run.smithi172.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2019-12-21T15:43:42.025 INFO:teuthology.orchestra.run.smithi172.stdout: python3-logutils-0.3.5-11.el8.noarch 2019-12-21T15:43:42.025 INFO:teuthology.orchestra.run.smithi172.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2019-12-21T15:43:42.025 INFO:teuthology.orchestra.run.smithi172.stdout: python3-pecan-1.3.2-9.el8.noarch 2019-12-21T15:43:42.025 INFO:teuthology.orchestra.run.smithi172.stdout: python3-portend-2.6-1.el8.noarch 2019-12-21T15:43:42.025 INFO:teuthology.orchestra.run.smithi172.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2019-12-21T15:43:42.026 INFO:teuthology.orchestra.run.smithi172.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2019-12-21T15:43:42.026 INFO:teuthology.orchestra.run.smithi172.stdout: python3-tempora-1.14.1-5.el8.noarch 2019-12-21T15:43:42.026 INFO:teuthology.orchestra.run.smithi172.stdout: python3-trustme-0.5.2-4.el8.noarch 2019-12-21T15:43:42.026 INFO:teuthology.orchestra.run.smithi172.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2019-12-21T15:43:42.026 INFO:teuthology.orchestra.run.smithi172.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2019-12-21T15:43:42.026 INFO:teuthology.orchestra.run.smithi172.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2019-12-21T15:43:42.026 INFO:teuthology.orchestra.run.smithi172.stdout: python3-waitress-1.2.1-2.el8.1.noarch 2019-12-21T15:43:42.027 INFO:teuthology.orchestra.run.smithi172.stdout: python3-webob-1.8.5-1.el8.1.noarch 2019-12-21T15:43:42.027 INFO:teuthology.orchestra.run.smithi172.stdout: python3-webtest-2.0.33-1.el8.noarch 2019-12-21T15:43:42.027 INFO:teuthology.orchestra.run.smithi172.stdout: python3-webencodings-0.5.1-6.el8.noarch 2019-12-21T15:43:42.027 INFO:teuthology.orchestra.run.smithi172.stdout: python3-lxml-4.2.3-1.el8.x86_64 2019-12-21T15:43:42.027 INFO:teuthology.orchestra.run.smithi172.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2019-12-21T15:43:42.027 INFO:teuthology.orchestra.run.smithi172.stdout: python3-mako-1.0.6-13.el8.noarch 2019-12-21T15:43:42.027 INFO:teuthology.orchestra.run.smithi172.stdout: python3-pytz-2017.2-9.el8.noarch 2019-12-21T15:43:42.028 INFO:teuthology.orchestra.run.smithi172.stdout: python3-markupsafe-0.23-19.el8.x86_64 2019-12-21T15:43:42.028 INFO:teuthology.orchestra.run.smithi172.stdout: 2019-12-21T15:43:42.028 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2019-12-21T15:43:42.189 INFO:teuthology.orchestra.run.smithi172:> sudo yum -y install ceph-base 2019-12-21T15:43:42.533 INFO:teuthology.orchestra.run.smithi172.stdout:Updating Subscription Management repositories. 2019-12-21T15:43:43.312 INFO:teuthology.orchestra.run.smithi172.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 91 kB/s | 2.8 kB 00:00 2019-12-21T15:43:43.375 INFO:teuthology.orchestra.run.smithi172.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 68 kB/s | 2.3 kB 00:00 2019-12-21T15:43:43.789 INFO:teuthology.orchestra.run.smithi172.stdout:Package ceph-base-2:15.0.0-8795.gc6e9f8f.el8.x86_64 is already installed. 2019-12-21T15:43:43.903 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2019-12-21T15:43:43.904 INFO:teuthology.orchestra.run.smithi172.stdout:Nothing to do. 2019-12-21T15:43:43.905 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2019-12-21T15:43:44.043 INFO:teuthology.orchestra.run.smithi172:> sudo yum -y install cephadm 2019-12-21T15:43:44.386 INFO:teuthology.orchestra.run.smithi172.stdout:Updating Subscription Management repositories. 2019-12-21T15:43:45.069 INFO:teuthology.orchestra.run.smithi172.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 90 kB/s | 2.8 kB 00:00 2019-12-21T15:43:45.128 INFO:teuthology.orchestra.run.smithi172.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 77 kB/s | 2.3 kB 00:00 2019-12-21T15:43:45.654 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2019-12-21T15:43:45.654 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2019-12-21T15:43:45.655 INFO:teuthology.orchestra.run.smithi172.stdout: Package Arch Version Repository Size 2019-12-21T15:43:45.655 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2019-12-21T15:43:45.655 INFO:teuthology.orchestra.run.smithi172.stdout:Installing: 2019-12-21T15:43:45.655 INFO:teuthology.orchestra.run.smithi172.stdout: cephadm x86_64 2:15.0.0-8795.gc6e9f8f.el8 ceph 27 k 2019-12-21T15:43:45.655 INFO:teuthology.orchestra.run.smithi172.stdout: 2019-12-21T15:43:45.655 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction Summary 2019-12-21T15:43:45.656 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2019-12-21T15:43:45.656 INFO:teuthology.orchestra.run.smithi172.stdout:Install 1 Package 2019-12-21T15:43:45.656 INFO:teuthology.orchestra.run.smithi172.stdout: 2019-12-21T15:43:45.656 INFO:teuthology.orchestra.run.smithi172.stdout:Total download size: 27 k 2019-12-21T15:43:45.657 INFO:teuthology.orchestra.run.smithi172.stdout:Installed size: 88 k 2019-12-21T15:43:45.657 INFO:teuthology.orchestra.run.smithi172.stdout:Downloading Packages: 2019-12-21T15:43:45.766 INFO:teuthology.orchestra.run.smithi172.stdout:cephadm-15.0.0-8795.gc6e9f8f.el8.x86_64.rpm 242 kB/s | 27 kB 00:00 2019-12-21T15:43:45.767 INFO:teuthology.orchestra.run.smithi172.stdout:-------------------------------------------------------------------------------- 2019-12-21T15:43:45.767 INFO:teuthology.orchestra.run.smithi172.stdout:Total 240 kB/s | 27 kB 00:00 2019-12-21T15:43:45.768 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction check 2019-12-21T15:43:45.775 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction check succeeded. 2019-12-21T15:43:45.775 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction test 2019-12-21T15:43:45.777 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction test succeeded. 2019-12-21T15:43:45.777 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction 2019-12-21T15:43:45.817 INFO:teuthology.orchestra.run.smithi172.stdout: Preparing : 1/1 2019-12-21T15:43:47.064 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: cephadm-2:15.0.0-8795.gc6e9f8f.el8.x86_64 1/1 2019-12-21T15:43:47.320 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : cephadm-2:15.0.0-8795.gc6e9f8f.el8.x86_64 1/1 2019-12-21T15:43:47.629 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : cephadm-2:15.0.0-8795.gc6e9f8f.el8.x86_64 1/1 2019-12-21T15:43:47.630 INFO:teuthology.orchestra.run.smithi172.stdout:Installed products updated. 2019-12-21T15:43:49.162 INFO:teuthology.orchestra.run.smithi172.stdout: 2019-12-21T15:43:49.162 INFO:teuthology.orchestra.run.smithi172.stdout:Installed: 2019-12-21T15:43:49.162 INFO:teuthology.orchestra.run.smithi172.stdout: cephadm-2:15.0.0-8795.gc6e9f8f.el8.x86_64 2019-12-21T15:43:49.162 INFO:teuthology.orchestra.run.smithi172.stdout: 2019-12-21T15:43:49.163 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2019-12-21T15:43:49.338 INFO:teuthology.orchestra.run.smithi172:> sudo yum -y install ceph-immutable-object-cache 2019-12-21T15:43:49.686 INFO:teuthology.orchestra.run.smithi172.stdout:Updating Subscription Management repositories. 2019-12-21T15:43:50.442 INFO:teuthology.orchestra.run.smithi172.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 89 kB/s | 2.8 kB 00:00 2019-12-21T15:43:50.528 INFO:teuthology.orchestra.run.smithi172.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 41 kB/s | 2.3 kB 00:00 2019-12-21T15:43:51.051 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2019-12-21T15:43:51.052 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2019-12-21T15:43:51.052 INFO:teuthology.orchestra.run.smithi172.stdout: Package Arch Version Repository 2019-12-21T15:43:51.052 INFO:teuthology.orchestra.run.smithi172.stdout: Size 2019-12-21T15:43:51.052 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2019-12-21T15:43:51.052 INFO:teuthology.orchestra.run.smithi172.stdout:Installing: 2019-12-21T15:43:51.053 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-immutable-object-cache x86_64 2:15.0.0-8795.gc6e9f8f.el8 ceph 195 k 2019-12-21T15:43:51.053 INFO:teuthology.orchestra.run.smithi172.stdout: 2019-12-21T15:43:51.053 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction Summary 2019-12-21T15:43:51.053 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2019-12-21T15:43:51.053 INFO:teuthology.orchestra.run.smithi172.stdout:Install 1 Package 2019-12-21T15:43:51.053 INFO:teuthology.orchestra.run.smithi172.stdout: 2019-12-21T15:43:51.054 INFO:teuthology.orchestra.run.smithi172.stdout:Total download size: 195 k 2019-12-21T15:43:51.054 INFO:teuthology.orchestra.run.smithi172.stdout:Installed size: 675 k 2019-12-21T15:43:51.054 INFO:teuthology.orchestra.run.smithi172.stdout:Downloading Packages: 2019-12-21T15:43:51.206 INFO:teuthology.orchestra.run.smithi172.stdout:ceph-immutable-object-cache-15.0.0-8795.gc6e9f8 1.2 MB/s | 195 kB 00:00 2019-12-21T15:43:51.207 INFO:teuthology.orchestra.run.smithi172.stdout:-------------------------------------------------------------------------------- 2019-12-21T15:43:51.207 INFO:teuthology.orchestra.run.smithi172.stdout:Total 1.2 MB/s | 195 kB 00:00 2019-12-21T15:43:51.207 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction check 2019-12-21T15:43:51.218 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction check succeeded. 2019-12-21T15:43:51.219 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction test 2019-12-21T15:43:51.264 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction test succeeded. 2019-12-21T15:43:51.264 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction 2019-12-21T15:43:51.421 INFO:teuthology.orchestra.run.smithi172.stdout: Preparing : 1/1 2019-12-21T15:43:51.601 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-immutable-object-cache-2:15.0.0-8795.gc6e9f8f.e 1/1 2019-12-21T15:43:51.908 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-immutable-object-cache-2:15.0.0-8795.gc6e9f8f.e 1/1 2019-12-21T15:43:52.239 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-immutable-object-cache-2:15.0.0-8795.gc6e9f8f.e 1/1 2019-12-21T15:43:52.239 INFO:teuthology.orchestra.run.smithi172.stdout:Installed products updated. 2019-12-21T15:43:53.718 INFO:teuthology.orchestra.run.smithi172.stdout: 2019-12-21T15:43:53.718 INFO:teuthology.orchestra.run.smithi172.stdout:Installed: 2019-12-21T15:43:53.719 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-immutable-object-cache-2:15.0.0-8795.gc6e9f8f.el8.x86_64 2019-12-21T15:43:53.719 INFO:teuthology.orchestra.run.smithi172.stdout: 2019-12-21T15:43:53.719 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2019-12-21T15:43:53.858 INFO:teuthology.orchestra.run.smithi172:> sudo yum -y install ceph-mgr 2019-12-21T15:43:54.199 INFO:teuthology.orchestra.run.smithi172.stdout:Updating Subscription Management repositories. 2019-12-21T15:43:54.865 INFO:teuthology.orchestra.run.smithi172.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 82 kB/s | 2.8 kB 00:00 2019-12-21T15:43:54.926 INFO:teuthology.orchestra.run.smithi172.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 73 kB/s | 2.3 kB 00:00 2019-12-21T15:43:55.326 INFO:teuthology.orchestra.run.smithi172.stdout:Package ceph-mgr-2:15.0.0-8795.gc6e9f8f.el8.x86_64 is already installed. 2019-12-21T15:43:55.440 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2019-12-21T15:43:55.441 INFO:teuthology.orchestra.run.smithi172.stdout:Nothing to do. 2019-12-21T15:43:55.441 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2019-12-21T15:43:55.604 INFO:teuthology.orchestra.run.smithi172:> sudo yum -y install ceph-mgr-dashboard 2019-12-21T15:43:55.943 INFO:teuthology.orchestra.run.smithi172.stdout:Updating Subscription Management repositories. 2019-12-21T15:43:56.533 INFO:teuthology.orchestra.run.smithi172.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 88 kB/s | 2.8 kB 00:00 2019-12-21T15:43:56.596 INFO:teuthology.orchestra.run.smithi172.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 72 kB/s | 2.3 kB 00:00 2019-12-21T15:43:57.143 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2019-12-21T15:43:57.144 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2019-12-21T15:43:57.144 INFO:teuthology.orchestra.run.smithi172.stdout: Package ArchVersion Repository Size 2019-12-21T15:43:57.144 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2019-12-21T15:43:57.145 INFO:teuthology.orchestra.run.smithi172.stdout:Installing: 2019-12-21T15:43:57.145 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mgr-dashboard noarch2:15.0.0-8795.gc6e9f8f.el8 ceph-noarch 3.8 M 2019-12-21T15:43:57.145 INFO:teuthology.orchestra.run.smithi172.stdout:Installing dependencies: 2019-12-21T15:43:57.145 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-grafana-dashboards noarch2:15.0.0-8795.gc6e9f8f.el8 ceph-noarch 22 k 2019-12-21T15:43:57.145 INFO:teuthology.orchestra.run.smithi172.stdout: python3-repoze-lru noarch0.7-6.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 34 k 2019-12-21T15:43:57.145 INFO:teuthology.orchestra.run.smithi172.stdout: python3-routes noarch2.4.1-12.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 197 k 2019-12-21T15:43:57.145 INFO:teuthology.orchestra.run.smithi172.stdout: python3-jwt noarch1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2019-12-21T15:43:57.146 INFO:teuthology.orchestra.run.smithi172.stdout: 2019-12-21T15:43:57.146 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction Summary 2019-12-21T15:43:57.146 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2019-12-21T15:43:57.146 INFO:teuthology.orchestra.run.smithi172.stdout:Install 5 Packages 2019-12-21T15:43:57.146 INFO:teuthology.orchestra.run.smithi172.stdout: 2019-12-21T15:43:57.147 INFO:teuthology.orchestra.run.smithi172.stdout:Total download size: 4.1 M 2019-12-21T15:43:57.147 INFO:teuthology.orchestra.run.smithi172.stdout:Installed size: 18 M 2019-12-21T15:43:57.147 INFO:teuthology.orchestra.run.smithi172.stdout:Downloading Packages: 2019-12-21T15:43:57.332 INFO:teuthology.orchestra.run.smithi172.stdout:(1/5): ceph-grafana-dashboards-15.0.0-8795.gc6e 121 kB/s | 22 kB 00:00 2019-12-21T15:43:57.521 INFO:teuthology.orchestra.run.smithi172.stdout:(2/5): python3-repoze-lru-0.7-6.el8.noarch.rpm 91 kB/s | 34 kB 00:00 2019-12-21T15:43:57.535 INFO:teuthology.orchestra.run.smithi172.stdout:(3/5): ceph-mgr-dashboard-15.0.0-8795.gc6e9f8f. 9.7 MB/s | 3.8 MB 00:00 2019-12-21T15:43:57.563 INFO:teuthology.orchestra.run.smithi172.stdout:(4/5): python3-jwt-1.6.1-2.el8.noarch.rpm 1.0 MB/s | 43 kB 00:00 2019-12-21T15:43:57.845 INFO:teuthology.orchestra.run.smithi172.stdout:(5/5): python3-routes-2.4.1-12.el8.noarch.rpm 383 kB/s | 197 kB 00:00 2019-12-21T15:43:57.848 INFO:teuthology.orchestra.run.smithi172.stdout:-------------------------------------------------------------------------------- 2019-12-21T15:43:57.848 INFO:teuthology.orchestra.run.smithi172.stdout:Total 5.8 MB/s | 4.1 MB 00:00 2019-12-21T15:43:57.850 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction check 2019-12-21T15:43:57.859 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction check succeeded. 2019-12-21T15:43:57.859 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction test 2019-12-21T15:43:57.967 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction test succeeded. 2019-12-21T15:43:57.967 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction 2019-12-21T15:43:58.201 INFO:teuthology.orchestra.run.smithi172.stdout: Preparing : 1/1 2019-12-21T15:43:58.433 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 1/5 2019-12-21T15:43:58.607 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 2/5 2019-12-21T15:43:58.790 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 3/5 2019-12-21T15:43:59.738 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-grafana-dashboards-2:15.0.0-8795.gc6e9f8f.el8.n 4/5 2019-12-21T15:43:59.999 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-mgr-dashboard-2:15.0.0-8795.gc6e9f8f.el8.noarch 5/5 2019-12-21T15:44:00.193 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-mgr-dashboard-2:15.0.0-8795.gc6e9f8f.el8.noarch 5/5 2019-12-21T15:44:00.193 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-grafana-dashboards-2:15.0.0-8795.gc6e9f8f.el8.n 1/5 2019-12-21T15:44:00.194 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-mgr-dashboard-2:15.0.0-8795.gc6e9f8f.el8.noarch 2/5 2019-12-21T15:44:00.194 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 3/5 2019-12-21T15:44:00.194 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 4/5 2019-12-21T15:44:00.548 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 5/5 2019-12-21T15:44:00.548 INFO:teuthology.orchestra.run.smithi172.stdout:Installed products updated. 2019-12-21T15:44:01.949 INFO:teuthology.orchestra.run.smithi172.stdout: 2019-12-21T15:44:01.949 INFO:teuthology.orchestra.run.smithi172.stdout:Installed: 2019-12-21T15:44:01.949 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mgr-dashboard-2:15.0.0-8795.gc6e9f8f.el8.noarch 2019-12-21T15:44:01.950 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-grafana-dashboards-2:15.0.0-8795.gc6e9f8f.el8.noarch 2019-12-21T15:44:01.950 INFO:teuthology.orchestra.run.smithi172.stdout: python3-repoze-lru-0.7-6.el8.noarch 2019-12-21T15:44:01.950 INFO:teuthology.orchestra.run.smithi172.stdout: python3-routes-2.4.1-12.el8.noarch 2019-12-21T15:44:01.950 INFO:teuthology.orchestra.run.smithi172.stdout: python3-jwt-1.6.1-2.el8.noarch 2019-12-21T15:44:01.950 INFO:teuthology.orchestra.run.smithi172.stdout: 2019-12-21T15:44:01.950 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2019-12-21T15:44:02.100 INFO:teuthology.orchestra.run.smithi172:> sudo yum -y install ceph-mgr-diskprediction-cloud 2019-12-21T15:44:02.456 INFO:teuthology.orchestra.run.smithi172.stdout:Updating Subscription Management repositories. 2019-12-21T15:44:02.991 INFO:teuthology.orchestra.run.smithi172.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 91 kB/s | 2.8 kB 00:00 2019-12-21T15:44:03.050 INFO:teuthology.orchestra.run.smithi172.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 77 kB/s | 2.3 kB 00:00 2019-12-21T15:44:03.570 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2019-12-21T15:44:03.570 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2019-12-21T15:44:03.570 INFO:teuthology.orchestra.run.smithi172.stdout: Package Arch Version Repository Size 2019-12-21T15:44:03.571 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2019-12-21T15:44:03.571 INFO:teuthology.orchestra.run.smithi172.stdout:Installing: 2019-12-21T15:44:03.571 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mgr-diskprediction-cloud 2019-12-21T15:44:03.571 INFO:teuthology.orchestra.run.smithi172.stdout: noarch 2:15.0.0-8795.gc6e9f8f.el8 ceph-noarch 41 k 2019-12-21T15:44:03.571 INFO:teuthology.orchestra.run.smithi172.stdout: 2019-12-21T15:44:03.572 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction Summary 2019-12-21T15:44:03.572 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2019-12-21T15:44:03.572 INFO:teuthology.orchestra.run.smithi172.stdout:Install 1 Package 2019-12-21T15:44:03.572 INFO:teuthology.orchestra.run.smithi172.stdout: 2019-12-21T15:44:03.573 INFO:teuthology.orchestra.run.smithi172.stdout:Total download size: 41 k 2019-12-21T15:44:03.573 INFO:teuthology.orchestra.run.smithi172.stdout:Installed size: 212 k 2019-12-21T15:44:03.573 INFO:teuthology.orchestra.run.smithi172.stdout:Downloading Packages: 2019-12-21T15:44:03.704 INFO:teuthology.orchestra.run.smithi172.stdout:ceph-mgr-diskprediction-cloud-15.0.0-8795.gc6e9 310 kB/s | 41 kB 00:00 2019-12-21T15:44:03.704 INFO:teuthology.orchestra.run.smithi172.stdout:-------------------------------------------------------------------------------- 2019-12-21T15:44:03.705 INFO:teuthology.orchestra.run.smithi172.stdout:Total 307 kB/s | 41 kB 00:00 2019-12-21T15:44:03.705 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction check 2019-12-21T15:44:03.709 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction check succeeded. 2019-12-21T15:44:03.709 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction test 2019-12-21T15:44:03.732 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction test succeeded. 2019-12-21T15:44:03.732 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction 2019-12-21T15:44:03.800 INFO:teuthology.orchestra.run.smithi172.stdout: Preparing : 1/1 2019-12-21T15:44:03.998 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-mgr-diskprediction-cloud-2:15.0.0-8795.gc6e9f8f 1/1 2019-12-21T15:44:04.123 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-mgr-diskprediction-cloud-2:15.0.0-8795.gc6e9f8f 1/1 2019-12-21T15:44:04.443 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-mgr-diskprediction-cloud-2:15.0.0-8795.gc6e9f8f 1/1 2019-12-21T15:44:04.443 INFO:teuthology.orchestra.run.smithi172.stdout:Installed products updated. 2019-12-21T15:44:05.822 INFO:teuthology.orchestra.run.smithi172.stdout: 2019-12-21T15:44:05.822 INFO:teuthology.orchestra.run.smithi172.stdout:Installed: 2019-12-21T15:44:05.822 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mgr-diskprediction-cloud-2:15.0.0-8795.gc6e9f8f.el8.noarch 2019-12-21T15:44:05.822 INFO:teuthology.orchestra.run.smithi172.stdout: 2019-12-21T15:44:05.822 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2019-12-21T15:44:05.977 INFO:teuthology.orchestra.run.smithi172:> sudo yum -y install ceph-mgr-diskprediction-local 2019-12-21T15:44:06.321 INFO:teuthology.orchestra.run.smithi172.stdout:Updating Subscription Management repositories. 2019-12-21T15:44:06.891 INFO:teuthology.orchestra.run.smithi172.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 91 kB/s | 2.8 kB 00:00 2019-12-21T15:44:06.941 INFO:teuthology.orchestra.run.smithi172.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 76 kB/s | 2.3 kB 00:00 2019-12-21T15:44:07.457 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2019-12-21T15:44:07.459 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2019-12-21T15:44:07.459 INFO:teuthology.orchestra.run.smithi172.stdout: Package ArchVersion Repository Size 2019-12-21T15:44:07.459 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2019-12-21T15:44:07.459 INFO:teuthology.orchestra.run.smithi172.stdout:Installing: 2019-12-21T15:44:07.459 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mgr-diskprediction-local noarch2:15.0.0-8795.gc6e9f8f.el8 ceph-noarch 8.6 M 2019-12-21T15:44:07.459 INFO:teuthology.orchestra.run.smithi172.stdout:Installing dependencies: 2019-12-21T15:44:07.460 INFO:teuthology.orchestra.run.smithi172.stdout: python3-numpy-f2py x86_641:1.14.3-6.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2019-12-21T15:44:07.460 INFO:teuthology.orchestra.run.smithi172.stdout: openblas-threads x86_640.3.3-2.el8 rhel-8-for-x86_64-appstream-rpms 4.4 M 2019-12-21T15:44:07.460 INFO:teuthology.orchestra.run.smithi172.stdout: python3-numpy x86_641:1.14.3-6.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2019-12-21T15:44:07.460 INFO:teuthology.orchestra.run.smithi172.stdout: openblas x86_640.3.3-2.el8 rhel-8-for-x86_64-appstream-rpms 4.3 M 2019-12-21T15:44:07.460 INFO:teuthology.orchestra.run.smithi172.stdout: python3-scipy x86_641.0.0-19.module+el8.0.0+2966+d39a1df3 rhel-8-for-x86_64-appstream-rpms 15 M 2019-12-21T15:44:07.460 INFO:teuthology.orchestra.run.smithi172.stdout: libquadmath x86_648.2.1-3.5.el8 rhel-8-for-x86_64-baseos-rpms 167 k 2019-12-21T15:44:07.460 INFO:teuthology.orchestra.run.smithi172.stdout: libgfortran x86_648.2.1-3.5.el8 rhel-8-for-x86_64-baseos-rpms 636 k 2019-12-21T15:44:07.461 INFO:teuthology.orchestra.run.smithi172.stdout: 2019-12-21T15:44:07.461 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction Summary 2019-12-21T15:44:07.461 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2019-12-21T15:44:07.461 INFO:teuthology.orchestra.run.smithi172.stdout:Install 8 Packages 2019-12-21T15:44:07.461 INFO:teuthology.orchestra.run.smithi172.stdout: 2019-12-21T15:44:07.462 INFO:teuthology.orchestra.run.smithi172.stdout:Total download size: 37 M 2019-12-21T15:44:07.462 INFO:teuthology.orchestra.run.smithi172.stdout:Installed size: 203 M 2019-12-21T15:44:07.462 INFO:teuthology.orchestra.run.smithi172.stdout:Downloading Packages: 2019-12-21T15:44:07.509 INFO:teuthology.orchestra.run.smithi172.stdout:(1/8): python3-numpy-f2py-1.14.3-6.el8.x86_64.r 4.6 MB/s | 225 kB 00:00 2019-12-21T15:44:07.555 INFO:teuthology.orchestra.run.smithi172.stdout:(2/8): openblas-threads-0.3.3-2.el8.x86_64.rpm 47 MB/s | 4.4 MB 00:00 2019-12-21T15:44:07.571 INFO:teuthology.orchestra.run.smithi172.stdout:(3/8): python3-numpy-1.14.3-6.el8.x86_64.rpm 61 MB/s | 3.7 MB 00:00 2019-12-21T15:44:07.618 INFO:teuthology.orchestra.run.smithi172.stdout:(4/8): openblas-0.3.3-2.el8.x86_64.rpm 67 MB/s | 4.3 MB 00:00 2019-12-21T15:44:07.641 INFO:teuthology.orchestra.run.smithi172.stdout:(5/8): libquadmath-8.2.1-3.5.el8.x86_64.rpm 7.4 MB/s | 167 kB 00:00 2019-12-21T15:44:07.665 INFO:teuthology.orchestra.run.smithi172.stdout:(6/8): libgfortran-8.2.1-3.5.el8.x86_64.rpm 26 MB/s | 636 kB 00:00 2019-12-21T15:44:07.752 INFO:teuthology.orchestra.run.smithi172.stdout:(7/8): python3-scipy-1.0.0-19.module+el8.0.0+29 80 MB/s | 15 MB 00:00 2019-12-21T15:44:07.961 INFO:teuthology.orchestra.run.smithi172.stdout:(8/8): ceph-mgr-diskprediction-local-15.0.0-879 17 MB/s | 8.6 MB 00:00 2019-12-21T15:44:07.962 INFO:teuthology.orchestra.run.smithi172.stdout:-------------------------------------------------------------------------------- 2019-12-21T15:44:07.963 INFO:teuthology.orchestra.run.smithi172.stdout:Total 73 MB/s | 37 MB 00:00 2019-12-21T15:44:07.972 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction check 2019-12-21T15:44:07.989 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction check succeeded. 2019-12-21T15:44:07.990 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction test 2019-12-21T15:44:08.272 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction test succeeded. 2019-12-21T15:44:08.272 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction 2019-12-21T15:44:08.778 INFO:teuthology.orchestra.run.smithi172.stdout: Preparing : 1/1 2019-12-21T15:44:08.964 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : libquadmath-8.2.1-3.5.el8.x86_64 1/8 2019-12-21T15:44:09.082 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: libquadmath-8.2.1-3.5.el8.x86_64 1/8 2019-12-21T15:44:09.256 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : libgfortran-8.2.1-3.5.el8.x86_64 2/8 2019-12-21T15:44:09.808 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: libgfortran-8.2.1-3.5.el8.x86_64 2/8 2019-12-21T15:44:09.973 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : openblas-threads-0.3.3-2.el8.x86_64 3/8 2019-12-21T15:44:10.510 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: openblas-threads-0.3.3-2.el8.x86_64 3/8 2019-12-21T15:44:10.681 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : openblas-0.3.3-2.el8.x86_64 4/8 2019-12-21T15:44:11.366 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: openblas-0.3.3-2.el8.x86_64 4/8 2019-12-21T15:44:11.686 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-numpy-1:1.14.3-6.el8.x86_64 5/8 2019-12-21T15:44:12.589 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-numpy-f2py-1:1.14.3-6.el8.x86_64 6/8 2019-12-21T15:44:14.542 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-6.el8.x86_64 6/8 2019-12-21T15:44:15.862 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-scipy-1.0.0-19.module+el8.0.0+2966+d39a1df3. 7/8 2019-12-21T15:44:16.051 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-mgr-diskprediction-local-2:15.0.0-8795.gc6e9f8f 8/8 2019-12-21T15:44:16.263 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:15.0.0-8795.gc6e9f8f 8/8 2019-12-21T15:44:16.263 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-mgr-diskprediction-local-2:15.0.0-8795.gc6e9f8f 1/8 2019-12-21T15:44:16.264 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-numpy-f2py-1:1.14.3-6.el8.x86_64 2/8 2019-12-21T15:44:16.264 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : openblas-threads-0.3.3-2.el8.x86_64 3/8 2019-12-21T15:44:16.264 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-numpy-1:1.14.3-6.el8.x86_64 4/8 2019-12-21T15:44:16.264 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : openblas-0.3.3-2.el8.x86_64 5/8 2019-12-21T15:44:16.264 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-scipy-1.0.0-19.module+el8.0.0+2966+d39a1df3. 6/8 2019-12-21T15:44:16.264 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : libquadmath-8.2.1-3.5.el8.x86_64 7/8 2019-12-21T15:44:16.637 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : libgfortran-8.2.1-3.5.el8.x86_64 8/8 2019-12-21T15:44:16.637 INFO:teuthology.orchestra.run.smithi172.stdout:Installed products updated. 2019-12-21T15:44:18.026 INFO:teuthology.orchestra.run.smithi172.stdout: 2019-12-21T15:44:18.026 INFO:teuthology.orchestra.run.smithi172.stdout:Installed: 2019-12-21T15:44:18.026 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mgr-diskprediction-local-2:15.0.0-8795.gc6e9f8f.el8.noarch 2019-12-21T15:44:18.026 INFO:teuthology.orchestra.run.smithi172.stdout: python3-numpy-f2py-1:1.14.3-6.el8.x86_64 2019-12-21T15:44:18.027 INFO:teuthology.orchestra.run.smithi172.stdout: openblas-threads-0.3.3-2.el8.x86_64 2019-12-21T15:44:18.027 INFO:teuthology.orchestra.run.smithi172.stdout: python3-numpy-1:1.14.3-6.el8.x86_64 2019-12-21T15:44:18.027 INFO:teuthology.orchestra.run.smithi172.stdout: openblas-0.3.3-2.el8.x86_64 2019-12-21T15:44:18.027 INFO:teuthology.orchestra.run.smithi172.stdout: python3-scipy-1.0.0-19.module+el8.0.0+2966+d39a1df3.x86_64 2019-12-21T15:44:18.027 INFO:teuthology.orchestra.run.smithi172.stdout: libquadmath-8.2.1-3.5.el8.x86_64 2019-12-21T15:44:18.027 INFO:teuthology.orchestra.run.smithi172.stdout: libgfortran-8.2.1-3.5.el8.x86_64 2019-12-21T15:44:18.027 INFO:teuthology.orchestra.run.smithi172.stdout: 2019-12-21T15:44:18.028 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2019-12-21T15:44:18.815 INFO:teuthology.orchestra.run.smithi172:> sudo yum -y install ceph-mgr-rook 2019-12-21T15:44:19.160 INFO:teuthology.orchestra.run.smithi172.stdout:Updating Subscription Management repositories. 2019-12-21T15:44:19.691 INFO:teuthology.orchestra.run.smithi172.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 84 kB/s | 2.8 kB 00:00 2019-12-21T15:44:19.753 INFO:teuthology.orchestra.run.smithi172.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 72 kB/s | 2.3 kB 00:00 2019-12-21T15:44:20.262 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2019-12-21T15:44:20.264 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2019-12-21T15:44:20.264 INFO:teuthology.orchestra.run.smithi172.stdout: Package ArchVersion Repository Size 2019-12-21T15:44:20.264 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2019-12-21T15:44:20.265 INFO:teuthology.orchestra.run.smithi172.stdout:Installing: 2019-12-21T15:44:20.265 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mgr-rook noarch2:15.0.0-8795.gc6e9f8f.el8 ceph-noarch 16 k 2019-12-21T15:44:20.265 INFO:teuthology.orchestra.run.smithi172.stdout:Installing dependencies: 2019-12-21T15:44:20.265 INFO:teuthology.orchestra.run.smithi172.stdout: python3-cachetools noarch3.1.1-4.el8 epel 33 k 2019-12-21T15:44:20.265 INFO:teuthology.orchestra.run.smithi172.stdout: python3-certifi noarch2018.10.15-7.el8 epel 16 k 2019-12-21T15:44:20.265 INFO:teuthology.orchestra.run.smithi172.stdout: python3-google-auth noarch1:1.1.1-10.el8 epel 81 k 2019-12-21T15:44:20.265 INFO:teuthology.orchestra.run.smithi172.stdout: python3-kubernetes noarch10.0.1-1.el8 epel 1.1 M 2019-12-21T15:44:20.266 INFO:teuthology.orchestra.run.smithi172.stdout: python3-rsa noarch3.4.2-12.el8 epel 73 k 2019-12-21T15:44:20.266 INFO:teuthology.orchestra.run.smithi172.stdout: python3-websocket-client noarch0.56.0-5.el8 epel 61 k 2019-12-21T15:44:20.266 INFO:teuthology.orchestra.run.smithi172.stdout: python3-pyasn1 noarch0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2019-12-21T15:44:20.266 INFO:teuthology.orchestra.run.smithi172.stdout: python3-pyasn1-modules noarch0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2019-12-21T15:44:20.266 INFO:teuthology.orchestra.run.smithi172.stdout: python3-requests-oauthlib noarch1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2019-12-21T15:44:20.266 INFO:teuthology.orchestra.run.smithi172.stdout: python3-oauthlib noarch2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2019-12-21T15:44:20.266 INFO:teuthology.orchestra.run.smithi172.stdout: 2019-12-21T15:44:20.267 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction Summary 2019-12-21T15:44:20.267 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2019-12-21T15:44:20.267 INFO:teuthology.orchestra.run.smithi172.stdout:Install 11 Packages 2019-12-21T15:44:20.267 INFO:teuthology.orchestra.run.smithi172.stdout: 2019-12-21T15:44:20.268 INFO:teuthology.orchestra.run.smithi172.stdout:Total download size: 1.8 M 2019-12-21T15:44:20.268 INFO:teuthology.orchestra.run.smithi172.stdout:Installed size: 23 M 2019-12-21T15:44:20.268 INFO:teuthology.orchestra.run.smithi172.stdout:Downloading Packages: 2019-12-21T15:44:20.447 INFO:teuthology.orchestra.run.smithi172.stdout:(1/11): ceph-mgr-rook-15.0.0-8795.gc6e9f8f.el8. 101 kB/s | 16 kB 00:00 2019-12-21T15:44:20.448 INFO:teuthology.orchestra.run.smithi172.stdout:(2/11): python3-certifi-2018.10.15-7.el8.noarch 101 kB/s | 16 kB 00:00 2019-12-21T15:44:20.448 INFO:teuthology.orchestra.run.smithi172.stdout:(3/11): python3-cachetools-3.1.1-4.el8.noarch.r 204 kB/s | 33 kB 00:00 2019-12-21T15:44:20.554 INFO:teuthology.orchestra.run.smithi172.stdout:(4/11): python3-google-auth-1.1.1-10.el8.noarch 764 kB/s | 81 kB 00:00 2019-12-21T15:44:20.608 INFO:teuthology.orchestra.run.smithi172.stdout:(5/11): python3-websocket-client-0.56.0-5.el8.n 1.1 MB/s | 61 kB 00:00 2019-12-21T15:44:20.647 INFO:teuthology.orchestra.run.smithi172.stdout:(6/11): python3-pyasn1-0.3.7-6.el8.noarch.rpm 3.2 MB/s | 126 kB 00:00 2019-12-21T15:44:20.662 INFO:teuthology.orchestra.run.smithi172.stdout:(7/11): python3-rsa-3.4.2-12.el8.noarch.rpm 340 kB/s | 73 kB 00:00 2019-12-21T15:44:20.671 INFO:teuthology.orchestra.run.smithi172.stdout:(8/11): python3-pyasn1-modules-0.3.7-6.el8.noar 4.6 MB/s | 110 kB 00:00 2019-12-21T15:44:20.702 INFO:teuthology.orchestra.run.smithi172.stdout:(9/11): python3-requests-oauthlib-1.0.0-1.el8.n 1.0 MB/s | 43 kB 00:00 2019-12-21T15:44:20.712 INFO:teuthology.orchestra.run.smithi172.stdout:(10/11): python3-oauthlib-2.1.0-1.el8.noarch.rp 3.7 MB/s | 155 kB 00:00 2019-12-21T15:44:20.721 INFO:teuthology.orchestra.run.smithi172.stdout:(11/11): python3-kubernetes-10.0.1-1.el8.noarch 4.1 MB/s | 1.1 MB 00:00 2019-12-21T15:44:20.721 INFO:teuthology.orchestra.run.smithi172.stdout:-------------------------------------------------------------------------------- 2019-12-21T15:44:20.722 INFO:teuthology.orchestra.run.smithi172.stdout:Total 4.0 MB/s | 1.8 MB 00:00 2019-12-21T15:44:20.725 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction check 2019-12-21T15:44:20.738 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction check succeeded. 2019-12-21T15:44:20.738 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction test 2019-12-21T15:44:20.789 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction test succeeded. 2019-12-21T15:44:20.790 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction 2019-12-21T15:44:21.172 INFO:teuthology.orchestra.run.smithi172.stdout: Preparing : 1/1 2019-12-21T15:44:21.368 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/11 2019-12-21T15:44:21.568 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-rsa-3.4.2-12.el8.noarch 2/11 2019-12-21T15:44:21.766 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 3/11 2019-12-21T15:44:21.967 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 4/11 2019-12-21T15:44:22.173 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 5/11 2019-12-21T15:44:22.361 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 6/11 2019-12-21T15:44:22.565 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 7/11 2019-12-21T15:44:22.808 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 8/11 2019-12-21T15:44:23.840 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 9/11 2019-12-21T15:44:24.131 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-kubernetes-10.0.1-1.el8.noarch 10/11 2019-12-21T15:44:24.312 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-mgr-rook-2:15.0.0-8795.gc6e9f8f.el8.noarch 11/11 2019-12-21T15:44:24.511 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-mgr-rook-2:15.0.0-8795.gc6e9f8f.el8.noarch 11/11 2019-12-21T15:44:24.511 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-mgr-rook-2:15.0.0-8795.gc6e9f8f.el8.noarch 1/11 2019-12-21T15:44:24.511 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 2/11 2019-12-21T15:44:24.511 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 3/11 2019-12-21T15:44:24.511 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 4/11 2019-12-21T15:44:24.512 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-kubernetes-10.0.1-1.el8.noarch 5/11 2019-12-21T15:44:24.512 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-rsa-3.4.2-12.el8.noarch 6/11 2019-12-21T15:44:24.512 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 7/11 2019-12-21T15:44:24.512 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 8/11 2019-12-21T15:44:24.512 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 9/11 2019-12-21T15:44:24.513 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 10/11 2019-12-21T15:44:24.929 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 11/11 2019-12-21T15:44:24.929 INFO:teuthology.orchestra.run.smithi172.stdout:Installed products updated. 2019-12-21T15:44:26.436 INFO:teuthology.orchestra.run.smithi172.stdout: 2019-12-21T15:44:26.436 INFO:teuthology.orchestra.run.smithi172.stdout:Installed: 2019-12-21T15:44:26.436 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mgr-rook-2:15.0.0-8795.gc6e9f8f.el8.noarch 2019-12-21T15:44:26.437 INFO:teuthology.orchestra.run.smithi172.stdout: python3-cachetools-3.1.1-4.el8.noarch 2019-12-21T15:44:26.437 INFO:teuthology.orchestra.run.smithi172.stdout: python3-certifi-2018.10.15-7.el8.noarch 2019-12-21T15:44:26.437 INFO:teuthology.orchestra.run.smithi172.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2019-12-21T15:44:26.437 INFO:teuthology.orchestra.run.smithi172.stdout: python3-kubernetes-10.0.1-1.el8.noarch 2019-12-21T15:44:26.437 INFO:teuthology.orchestra.run.smithi172.stdout: python3-rsa-3.4.2-12.el8.noarch 2019-12-21T15:44:26.437 INFO:teuthology.orchestra.run.smithi172.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2019-12-21T15:44:26.438 INFO:teuthology.orchestra.run.smithi172.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2019-12-21T15:44:26.438 INFO:teuthology.orchestra.run.smithi172.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2019-12-21T15:44:26.438 INFO:teuthology.orchestra.run.smithi172.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2019-12-21T15:44:26.438 INFO:teuthology.orchestra.run.smithi172.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2019-12-21T15:44:26.438 INFO:teuthology.orchestra.run.smithi172.stdout: 2019-12-21T15:44:26.438 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2019-12-21T15:44:26.603 INFO:teuthology.orchestra.run.smithi172:> sudo yum -y install ceph-mgr-cephadm 2019-12-21T15:44:26.952 INFO:teuthology.orchestra.run.smithi172.stdout:Updating Subscription Management repositories. 2019-12-21T15:44:27.500 INFO:teuthology.orchestra.run.smithi172.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 91 kB/s | 2.8 kB 00:00 2019-12-21T15:44:27.560 INFO:teuthology.orchestra.run.smithi172.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 77 kB/s | 2.3 kB 00:00 2019-12-21T15:44:28.080 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2019-12-21T15:44:28.081 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2019-12-21T15:44:28.081 INFO:teuthology.orchestra.run.smithi172.stdout: Package ArchVersion Repository Size 2019-12-21T15:44:28.081 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2019-12-21T15:44:28.081 INFO:teuthology.orchestra.run.smithi172.stdout:Installing: 2019-12-21T15:44:28.081 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mgr-cephadm noarch2:15.0.0-8795.gc6e9f8f.el8 ceph-noarch 23 k 2019-12-21T15:44:28.081 INFO:teuthology.orchestra.run.smithi172.stdout:Installing dependencies: 2019-12-21T15:44:28.082 INFO:teuthology.orchestra.run.smithi172.stdout: python3-execnet noarch1.7.1-1.el8.2 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 356 k 2019-12-21T15:44:28.082 INFO:teuthology.orchestra.run.smithi172.stdout: python3-remoto noarch1.1.4-4.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 48 k 2019-12-21T15:44:28.082 INFO:teuthology.orchestra.run.smithi172.stdout: python3-apipkg noarch1.5-2.el8 epel 21 k 2019-12-21T15:44:28.082 INFO:teuthology.orchestra.run.smithi172.stdout: 2019-12-21T15:44:28.082 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction Summary 2019-12-21T15:44:28.082 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2019-12-21T15:44:28.082 INFO:teuthology.orchestra.run.smithi172.stdout:Install 4 Packages 2019-12-21T15:44:28.083 INFO:teuthology.orchestra.run.smithi172.stdout: 2019-12-21T15:44:28.083 INFO:teuthology.orchestra.run.smithi172.stdout:Total download size: 448 k 2019-12-21T15:44:28.083 INFO:teuthology.orchestra.run.smithi172.stdout:Installed size: 1.4 M 2019-12-21T15:44:28.083 INFO:teuthology.orchestra.run.smithi172.stdout:Downloading Packages: 2019-12-21T15:44:29.308 INFO:teuthology.orchestra.run.smithi172.stdout:(1/4): ceph-mgr-cephadm-15.0.0-8795.gc6e9f8f.el 164 kB/s | 23 kB 00:00 2019-12-21T15:44:29.432 INFO:teuthology.orchestra.run.smithi172.stdout:(2/4): python3-apipkg-1.5-2.el8.noarch.rpm 169 kB/s | 21 kB 00:00 2019-12-21T15:44:29.554 INFO:teuthology.orchestra.run.smithi172.stdout:(3/4): python3-remoto-1.1.4-4.el8.noarch.rpm 124 kB/s | 48 kB 00:00 2019-12-21T15:44:29.622 INFO:teuthology.orchestra.run.smithi172.stdout:(4/4): python3-execnet-1.7.1-1.el8.2.noarch.rpm 778 kB/s | 356 kB 00:00 2019-12-21T15:44:29.625 INFO:teuthology.orchestra.run.smithi172.stdout:-------------------------------------------------------------------------------- 2019-12-21T15:44:29.626 INFO:teuthology.orchestra.run.smithi172.stdout:Total 290 kB/s | 448 kB 00:01 2019-12-21T15:44:29.628 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction check 2019-12-21T15:44:29.636 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction check succeeded. 2019-12-21T15:44:29.636 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction test 2019-12-21T15:44:29.661 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction test succeeded. 2019-12-21T15:44:29.662 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction 2019-12-21T15:44:29.825 INFO:teuthology.orchestra.run.smithi172.stdout: Preparing : 1/1 2019-12-21T15:44:30.093 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-apipkg-1.5-2.el8.noarch 1/4 2019-12-21T15:44:30.349 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-execnet-1.7.1-1.el8.2.noarch 2/4 2019-12-21T15:44:30.574 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-remoto-1.1.4-4.el8.noarch 3/4 2019-12-21T15:44:30.711 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-mgr-cephadm-2:15.0.0-8795.gc6e9f8f.el8.noarch 4/4 2019-12-21T15:44:30.907 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-mgr-cephadm-2:15.0.0-8795.gc6e9f8f.el8.noarch 4/4 2019-12-21T15:44:30.907 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-mgr-cephadm-2:15.0.0-8795.gc6e9f8f.el8.noarch 1/4 2019-12-21T15:44:30.908 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-execnet-1.7.1-1.el8.2.noarch 2/4 2019-12-21T15:44:30.908 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-remoto-1.1.4-4.el8.noarch 3/4 2019-12-21T15:44:31.251 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-apipkg-1.5-2.el8.noarch 4/4 2019-12-21T15:44:31.252 INFO:teuthology.orchestra.run.smithi172.stdout:Installed products updated. 2019-12-21T15:44:32.740 INFO:teuthology.orchestra.run.smithi172.stdout: 2019-12-21T15:44:32.740 INFO:teuthology.orchestra.run.smithi172.stdout:Installed: 2019-12-21T15:44:32.740 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mgr-cephadm-2:15.0.0-8795.gc6e9f8f.el8.noarch 2019-12-21T15:44:32.741 INFO:teuthology.orchestra.run.smithi172.stdout: python3-execnet-1.7.1-1.el8.2.noarch 2019-12-21T15:44:32.741 INFO:teuthology.orchestra.run.smithi172.stdout: python3-remoto-1.1.4-4.el8.noarch 2019-12-21T15:44:32.741 INFO:teuthology.orchestra.run.smithi172.stdout: python3-apipkg-1.5-2.el8.noarch 2019-12-21T15:44:32.741 INFO:teuthology.orchestra.run.smithi172.stdout: 2019-12-21T15:44:32.741 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2019-12-21T15:44:32.895 INFO:teuthology.orchestra.run.smithi172:> sudo yum -y install ceph-fuse 2019-12-21T15:44:33.248 INFO:teuthology.orchestra.run.smithi172.stdout:Updating Subscription Management repositories. 2019-12-21T15:44:33.770 INFO:teuthology.orchestra.run.smithi172.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 87 kB/s | 2.8 kB 00:00 2019-12-21T15:44:33.830 INFO:teuthology.orchestra.run.smithi172.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 77 kB/s | 2.3 kB 00:00 2019-12-21T15:44:34.329 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2019-12-21T15:44:34.330 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2019-12-21T15:44:34.330 INFO:teuthology.orchestra.run.smithi172.stdout: Package Arch Version Repository Size 2019-12-21T15:44:34.330 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2019-12-21T15:44:34.330 INFO:teuthology.orchestra.run.smithi172.stdout:Installing: 2019-12-21T15:44:34.330 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-fuse x86_64 2:15.0.0-8795.gc6e9f8f.el8 2019-12-21T15:44:34.331 INFO:teuthology.orchestra.run.smithi172.stdout: ceph 567 k 2019-12-21T15:44:34.331 INFO:teuthology.orchestra.run.smithi172.stdout:Installing dependencies: 2019-12-21T15:44:34.331 INFO:teuthology.orchestra.run.smithi172.stdout: fuse x86_64 2.9.7-12.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2019-12-21T15:44:34.331 INFO:teuthology.orchestra.run.smithi172.stdout: fuse-common x86_64 3.2.1-12.el8 rhel-8-for-x86_64-baseos-rpms 21 k 2019-12-21T15:44:34.331 INFO:teuthology.orchestra.run.smithi172.stdout: 2019-12-21T15:44:34.331 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction Summary 2019-12-21T15:44:34.332 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2019-12-21T15:44:34.332 INFO:teuthology.orchestra.run.smithi172.stdout:Install 3 Packages 2019-12-21T15:44:34.332 INFO:teuthology.orchestra.run.smithi172.stdout: 2019-12-21T15:44:34.332 INFO:teuthology.orchestra.run.smithi172.stdout:Total download size: 670 k 2019-12-21T15:44:34.333 INFO:teuthology.orchestra.run.smithi172.stdout:Installed size: 2.0 M 2019-12-21T15:44:34.333 INFO:teuthology.orchestra.run.smithi172.stdout:Downloading Packages: 2019-12-21T15:44:34.383 INFO:teuthology.orchestra.run.smithi172.stdout:(1/3): fuse-2.9.7-12.el8.x86_64.rpm 1.5 MB/s | 83 kB 00:00 2019-12-21T15:44:34.385 INFO:teuthology.orchestra.run.smithi172.stdout:(2/3): fuse-common-3.2.1-12.el8.x86_64.rpm 392 kB/s | 21 kB 00:00 2019-12-21T15:44:34.531 INFO:teuthology.orchestra.run.smithi172.stdout:(3/3): ceph-fuse-15.0.0-8795.gc6e9f8f.el8.x86_6 2.8 MB/s | 567 kB 00:00 2019-12-21T15:44:34.531 INFO:teuthology.orchestra.run.smithi172.stdout:-------------------------------------------------------------------------------- 2019-12-21T15:44:34.532 INFO:teuthology.orchestra.run.smithi172.stdout:Total 3.3 MB/s | 670 kB 00:00 2019-12-21T15:44:34.534 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction check 2019-12-21T15:44:34.554 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction check succeeded. 2019-12-21T15:44:34.554 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction test 2019-12-21T15:44:34.612 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction test succeeded. 2019-12-21T15:44:34.612 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction 2019-12-21T15:44:34.777 INFO:teuthology.orchestra.run.smithi172.stdout: Preparing : 1/1 2019-12-21T15:44:34.918 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : fuse-common-3.2.1-12.el8.x86_64 1/3 2019-12-21T15:44:34.919 INFO:teuthology.orchestra.run.smithi172.stdout:warning: /etc/fuse.conf created as /etc/fuse.conf.rpmnew 2019-12-21T15:44:34.919 INFO:teuthology.orchestra.run.smithi172.stdout: 2019-12-21T15:44:35.150 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : fuse-2.9.7-12.el8.x86_64 2/3 2019-12-21T15:44:35.350 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-fuse-2:15.0.0-8795.gc6e9f8f.el8.x86_64 3/3 2019-12-21T15:44:35.628 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-fuse-2:15.0.0-8795.gc6e9f8f.el8.x86_64 3/3 2019-12-21T15:44:35.629 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-fuse-2:15.0.0-8795.gc6e9f8f.el8.x86_64 1/3 2019-12-21T15:44:35.629 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : fuse-2.9.7-12.el8.x86_64 2/3 2019-12-21T15:44:36.008 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : fuse-common-3.2.1-12.el8.x86_64 3/3 2019-12-21T15:44:36.009 INFO:teuthology.orchestra.run.smithi172.stdout:Installed products updated. 2019-12-21T15:44:37.511 INFO:teuthology.orchestra.run.smithi172.stdout: 2019-12-21T15:44:37.511 INFO:teuthology.orchestra.run.smithi172.stdout:Installed: 2019-12-21T15:44:37.511 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-fuse-2:15.0.0-8795.gc6e9f8f.el8.x86_64 fuse-2.9.7-12.el8.x86_64 2019-12-21T15:44:37.511 INFO:teuthology.orchestra.run.smithi172.stdout: fuse-common-3.2.1-12.el8.x86_64 2019-12-21T15:44:37.512 INFO:teuthology.orchestra.run.smithi172.stdout: 2019-12-21T15:44:37.512 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2019-12-21T15:44:37.673 INFO:teuthology.orchestra.run.smithi172:> sudo yum -y install librados-devel 2019-12-21T15:44:38.024 INFO:teuthology.orchestra.run.smithi172.stdout:Updating Subscription Management repositories. 2019-12-21T15:44:38.634 INFO:teuthology.orchestra.run.smithi172.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 91 kB/s | 2.8 kB 00:00 2019-12-21T15:44:38.702 INFO:teuthology.orchestra.run.smithi172.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 61 kB/s | 2.3 kB 00:00 2019-12-21T15:44:39.220 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2019-12-21T15:44:39.221 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2019-12-21T15:44:39.221 INFO:teuthology.orchestra.run.smithi172.stdout: Package Arch Version Repository 2019-12-21T15:44:39.221 INFO:teuthology.orchestra.run.smithi172.stdout: Size 2019-12-21T15:44:39.221 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2019-12-21T15:44:39.221 INFO:teuthology.orchestra.run.smithi172.stdout:Installing: 2019-12-21T15:44:39.221 INFO:teuthology.orchestra.run.smithi172.stdout: librados-devel x86_64 2:15.0.0-8795.gc6e9f8f.el8 ceph 139 k 2019-12-21T15:44:39.222 INFO:teuthology.orchestra.run.smithi172.stdout: 2019-12-21T15:44:39.222 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction Summary 2019-12-21T15:44:39.222 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2019-12-21T15:44:39.222 INFO:teuthology.orchestra.run.smithi172.stdout:Install 1 Package 2019-12-21T15:44:39.222 INFO:teuthology.orchestra.run.smithi172.stdout: 2019-12-21T15:44:39.222 INFO:teuthology.orchestra.run.smithi172.stdout:Total download size: 139 k 2019-12-21T15:44:39.222 INFO:teuthology.orchestra.run.smithi172.stdout:Installed size: 530 k 2019-12-21T15:44:39.223 INFO:teuthology.orchestra.run.smithi172.stdout:Downloading Packages: 2019-12-21T15:44:39.376 INFO:teuthology.orchestra.run.smithi172.stdout:librados-devel-15.0.0-8795.gc6e9f8f.el8.x86_64. 912 kB/s | 139 kB 00:00 2019-12-21T15:44:39.376 INFO:teuthology.orchestra.run.smithi172.stdout:-------------------------------------------------------------------------------- 2019-12-21T15:44:39.376 INFO:teuthology.orchestra.run.smithi172.stdout:Total 905 kB/s | 139 kB 00:00 2019-12-21T15:44:39.377 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction check 2019-12-21T15:44:39.381 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction check succeeded. 2019-12-21T15:44:39.381 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction test 2019-12-21T15:44:39.431 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction test succeeded. 2019-12-21T15:44:39.431 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction 2019-12-21T15:44:39.785 INFO:teuthology.orchestra.run.smithi172.stdout: Preparing : 1/1 2019-12-21T15:44:39.965 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : librados-devel-2:15.0.0-8795.gc6e9f8f.el8.x86_64 1/1 2019-12-21T15:44:40.159 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: librados-devel-2:15.0.0-8795.gc6e9f8f.el8.x86_64 1/1 2019-12-21T15:44:40.528 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : librados-devel-2:15.0.0-8795.gc6e9f8f.el8.x86_64 1/1 2019-12-21T15:44:40.529 INFO:teuthology.orchestra.run.smithi172.stdout:Installed products updated. 2019-12-21T15:44:42.039 INFO:teuthology.orchestra.run.smithi172.stdout: 2019-12-21T15:44:42.040 INFO:teuthology.orchestra.run.smithi172.stdout:Installed: 2019-12-21T15:44:42.040 INFO:teuthology.orchestra.run.smithi172.stdout: librados-devel-2:15.0.0-8795.gc6e9f8f.el8.x86_64 2019-12-21T15:44:42.040 INFO:teuthology.orchestra.run.smithi172.stdout: 2019-12-21T15:44:42.040 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2019-12-21T15:44:42.196 INFO:teuthology.orchestra.run.smithi172:> sudo yum -y install libcephfs2 2019-12-21T15:44:42.546 INFO:teuthology.orchestra.run.smithi172.stdout:Updating Subscription Management repositories. 2019-12-21T15:44:43.135 INFO:teuthology.orchestra.run.smithi172.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 88 kB/s | 2.8 kB 00:00 2019-12-21T15:44:43.194 INFO:teuthology.orchestra.run.smithi172.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 80 kB/s | 2.3 kB 00:00 2019-12-21T15:44:43.597 INFO:teuthology.orchestra.run.smithi172.stdout:Package libcephfs2-2:15.0.0-8795.gc6e9f8f.el8.x86_64 is already installed. 2019-12-21T15:44:43.715 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2019-12-21T15:44:43.716 INFO:teuthology.orchestra.run.smithi172.stdout:Nothing to do. 2019-12-21T15:44:43.716 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2019-12-21T15:44:43.865 INFO:teuthology.orchestra.run.smithi172:> sudo yum -y install libcephfs-devel 2019-12-21T15:44:44.213 INFO:teuthology.orchestra.run.smithi172.stdout:Updating Subscription Management repositories. 2019-12-21T15:44:44.820 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:00:01 ago on Sat 21 Dec 2019 03:44:43 PM UTC. 2019-12-21T15:44:45.780 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2019-12-21T15:44:45.781 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2019-12-21T15:44:45.781 INFO:teuthology.orchestra.run.smithi172.stdout: Package Arch Version Repository 2019-12-21T15:44:45.782 INFO:teuthology.orchestra.run.smithi172.stdout: Size 2019-12-21T15:44:45.782 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2019-12-21T15:44:45.782 INFO:teuthology.orchestra.run.smithi172.stdout:Installing: 2019-12-21T15:44:45.782 INFO:teuthology.orchestra.run.smithi172.stdout: libcephfs-devel x86_64 2:15.0.0-8795.gc6e9f8f.el8 ceph 21 k 2019-12-21T15:44:45.782 INFO:teuthology.orchestra.run.smithi172.stdout: 2019-12-21T15:44:45.782 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction Summary 2019-12-21T15:44:45.782 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2019-12-21T15:44:45.783 INFO:teuthology.orchestra.run.smithi172.stdout:Install 1 Package 2019-12-21T15:44:45.783 INFO:teuthology.orchestra.run.smithi172.stdout: 2019-12-21T15:44:45.783 INFO:teuthology.orchestra.run.smithi172.stdout:Total download size: 21 k 2019-12-21T15:44:45.783 INFO:teuthology.orchestra.run.smithi172.stdout:Installed size: 75 k 2019-12-21T15:44:45.783 INFO:teuthology.orchestra.run.smithi172.stdout:Downloading Packages: 2019-12-21T15:44:48.991 INFO:teuthology.orchestra.run.smithi172.stdout:libcephfs-devel-15.0.0-8795.gc6e9f8f.el8.x86_64 6.6 kB/s | 21 kB 00:03 2019-12-21T15:44:48.991 INFO:teuthology.orchestra.run.smithi172.stdout:-------------------------------------------------------------------------------- 2019-12-21T15:44:48.992 INFO:teuthology.orchestra.run.smithi172.stdout:Total 6.6 kB/s | 21 kB 00:03 2019-12-21T15:44:48.992 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction check 2019-12-21T15:44:48.996 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction check succeeded. 2019-12-21T15:44:48.997 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction test 2019-12-21T15:44:48.999 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction test succeeded. 2019-12-21T15:44:49.000 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction 2019-12-21T15:44:49.047 INFO:teuthology.orchestra.run.smithi172.stdout: Preparing : 1/1 2019-12-21T15:44:49.217 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : libcephfs-devel-2:15.0.0-8795.gc6e9f8f.el8.x86_64 1/1 2019-12-21T15:44:49.398 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: libcephfs-devel-2:15.0.0-8795.gc6e9f8f.el8.x86_64 1/1 2019-12-21T15:44:49.714 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : libcephfs-devel-2:15.0.0-8795.gc6e9f8f.el8.x86_64 1/1 2019-12-21T15:44:49.715 INFO:teuthology.orchestra.run.smithi172.stdout:Installed products updated. 2019-12-21T15:44:51.085 INFO:teuthology.orchestra.run.smithi172.stdout: 2019-12-21T15:44:51.086 INFO:teuthology.orchestra.run.smithi172.stdout:Installed: 2019-12-21T15:44:51.086 INFO:teuthology.orchestra.run.smithi172.stdout: libcephfs-devel-2:15.0.0-8795.gc6e9f8f.el8.x86_64 2019-12-21T15:44:51.086 INFO:teuthology.orchestra.run.smithi172.stdout: 2019-12-21T15:44:51.086 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2019-12-21T15:44:51.249 INFO:teuthology.orchestra.run.smithi172:> sudo yum -y install librados2 2019-12-21T15:44:51.597 INFO:teuthology.orchestra.run.smithi172.stdout:Updating Subscription Management repositories. 2019-12-21T15:44:52.160 INFO:teuthology.orchestra.run.smithi172.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 90 kB/s | 2.8 kB 00:00 2019-12-21T15:44:52.220 INFO:teuthology.orchestra.run.smithi172.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 77 kB/s | 2.3 kB 00:00 2019-12-21T15:44:52.623 INFO:teuthology.orchestra.run.smithi172.stdout:Package librados2-2:15.0.0-8795.gc6e9f8f.el8.x86_64 is already installed. 2019-12-21T15:44:52.738 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2019-12-21T15:44:52.740 INFO:teuthology.orchestra.run.smithi172.stdout:Nothing to do. 2019-12-21T15:44:52.740 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2019-12-21T15:44:52.898 INFO:teuthology.orchestra.run.smithi172:> sudo yum -y install librbd1 2019-12-21T15:44:53.248 INFO:teuthology.orchestra.run.smithi172.stdout:Updating Subscription Management repositories. 2019-12-21T15:44:53.832 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:00:01 ago on Sat 21 Dec 2019 03:44:52 PM UTC. 2019-12-21T15:44:54.206 INFO:teuthology.orchestra.run.smithi172.stdout:Package librbd1-2:15.0.0-8795.gc6e9f8f.el8.x86_64 is already installed. 2019-12-21T15:44:54.325 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2019-12-21T15:44:54.326 INFO:teuthology.orchestra.run.smithi172.stdout:Nothing to do. 2019-12-21T15:44:54.326 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2019-12-21T15:44:54.460 INFO:teuthology.orchestra.run.smithi172:> sudo yum -y install python3-rados 2019-12-21T15:44:54.811 INFO:teuthology.orchestra.run.smithi172.stdout:Updating Subscription Management repositories. 2019-12-21T15:44:55.617 INFO:teuthology.orchestra.run.smithi172.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 90 kB/s | 2.8 kB 00:00 2019-12-21T15:44:55.668 INFO:teuthology.orchestra.run.smithi172.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 72 kB/s | 2.3 kB 00:00 2019-12-21T15:44:56.092 INFO:teuthology.orchestra.run.smithi172.stdout:Package python3-rados-2:15.0.0-8795.gc6e9f8f.el8.x86_64 is already installed. 2019-12-21T15:44:56.209 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2019-12-21T15:44:56.210 INFO:teuthology.orchestra.run.smithi172.stdout:Nothing to do. 2019-12-21T15:44:56.210 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2019-12-21T15:44:56.348 INFO:teuthology.orchestra.run.smithi172:> sudo yum -y install python3-rgw 2019-12-21T15:44:56.694 INFO:teuthology.orchestra.run.smithi172.stdout:Updating Subscription Management repositories. 2019-12-21T15:44:57.247 INFO:teuthology.orchestra.run.smithi172.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 90 kB/s | 2.8 kB 00:00 2019-12-21T15:44:57.306 INFO:teuthology.orchestra.run.smithi172.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 78 kB/s | 2.3 kB 00:00 2019-12-21T15:44:57.708 INFO:teuthology.orchestra.run.smithi172.stdout:Package python3-rgw-2:15.0.0-8795.gc6e9f8f.el8.x86_64 is already installed. 2019-12-21T15:44:57.824 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2019-12-21T15:44:57.825 INFO:teuthology.orchestra.run.smithi172.stdout:Nothing to do. 2019-12-21T15:44:57.825 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2019-12-21T15:44:57.969 INFO:teuthology.orchestra.run.smithi172:> sudo yum -y install python3-cephfs 2019-12-21T15:44:58.317 INFO:teuthology.orchestra.run.smithi172.stdout:Updating Subscription Management repositories. 2019-12-21T15:44:58.969 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:00:01 ago on Sat 21 Dec 2019 03:44:57 PM UTC. 2019-12-21T15:44:59.322 INFO:teuthology.orchestra.run.smithi172.stdout:Package python3-cephfs-2:15.0.0-8795.gc6e9f8f.el8.x86_64 is already installed. 2019-12-21T15:44:59.439 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2019-12-21T15:44:59.440 INFO:teuthology.orchestra.run.smithi172.stdout:Nothing to do. 2019-12-21T15:44:59.440 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2019-12-21T15:44:59.567 INFO:teuthology.orchestra.run.smithi172:> sudo yum -y install python3-rbd 2019-12-21T15:44:59.916 INFO:teuthology.orchestra.run.smithi172.stdout:Updating Subscription Management repositories. 2019-12-21T15:45:00.830 INFO:teuthology.orchestra.run.smithi172.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 91 kB/s | 2.8 kB 00:00 2019-12-21T15:45:00.881 INFO:teuthology.orchestra.run.smithi172.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 74 kB/s | 2.3 kB 00:00 2019-12-21T15:45:01.291 INFO:teuthology.orchestra.run.smithi172.stdout:Package python3-rbd-2:15.0.0-8795.gc6e9f8f.el8.x86_64 is already installed. 2019-12-21T15:45:01.407 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2019-12-21T15:45:01.409 INFO:teuthology.orchestra.run.smithi172.stdout:Nothing to do. 2019-12-21T15:45:01.409 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2019-12-21T15:45:01.564 INFO:teuthology.orchestra.run.smithi172:> sudo yum -y install rbd-fuse 2019-12-21T15:45:01.912 INFO:teuthology.orchestra.run.smithi172.stdout:Updating Subscription Management repositories. 2019-12-21T15:45:02.430 INFO:teuthology.orchestra.run.smithi172.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 90 kB/s | 2.8 kB 00:00 2019-12-21T15:45:02.490 INFO:teuthology.orchestra.run.smithi172.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 77 kB/s | 2.3 kB 00:00 2019-12-21T15:45:03.024 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2019-12-21T15:45:03.025 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2019-12-21T15:45:03.025 INFO:teuthology.orchestra.run.smithi172.stdout: Package Arch Version Repository Size 2019-12-21T15:45:03.025 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2019-12-21T15:45:03.025 INFO:teuthology.orchestra.run.smithi172.stdout:Installing: 2019-12-21T15:45:03.025 INFO:teuthology.orchestra.run.smithi172.stdout: rbd-fuse x86_64 2:15.0.0-8795.gc6e9f8f.el8 ceph 72 k 2019-12-21T15:45:03.026 INFO:teuthology.orchestra.run.smithi172.stdout: 2019-12-21T15:45:03.026 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction Summary 2019-12-21T15:45:03.026 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2019-12-21T15:45:03.026 INFO:teuthology.orchestra.run.smithi172.stdout:Install 1 Package 2019-12-21T15:45:03.026 INFO:teuthology.orchestra.run.smithi172.stdout: 2019-12-21T15:45:03.026 INFO:teuthology.orchestra.run.smithi172.stdout:Total download size: 72 k 2019-12-21T15:45:03.026 INFO:teuthology.orchestra.run.smithi172.stdout:Installed size: 214 k 2019-12-21T15:45:03.027 INFO:teuthology.orchestra.run.smithi172.stdout:Downloading Packages: 2019-12-21T15:45:03.157 INFO:teuthology.orchestra.run.smithi172.stdout:rbd-fuse-15.0.0-8795.gc6e9f8f.el8.x86_64.rpm 548 kB/s | 72 kB 00:00 2019-12-21T15:45:03.158 INFO:teuthology.orchestra.run.smithi172.stdout:-------------------------------------------------------------------------------- 2019-12-21T15:45:03.158 INFO:teuthology.orchestra.run.smithi172.stdout:Total 543 kB/s | 72 kB 00:00 2019-12-21T15:45:03.158 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction check 2019-12-21T15:45:03.168 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction check succeeded. 2019-12-21T15:45:03.168 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction test 2019-12-21T15:45:03.215 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction test succeeded. 2019-12-21T15:45:03.215 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction 2019-12-21T15:45:03.309 INFO:teuthology.orchestra.run.smithi172.stdout: Preparing : 1/1 2019-12-21T15:45:03.494 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : rbd-fuse-2:15.0.0-8795.gc6e9f8f.el8.x86_64 1/1 2019-12-21T15:45:03.686 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: rbd-fuse-2:15.0.0-8795.gc6e9f8f.el8.x86_64 1/1 2019-12-21T15:45:04.034 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : rbd-fuse-2:15.0.0-8795.gc6e9f8f.el8.x86_64 1/1 2019-12-21T15:45:04.034 INFO:teuthology.orchestra.run.smithi172.stdout:Installed products updated. 2019-12-21T15:45:05.465 INFO:teuthology.orchestra.run.smithi172.stdout: 2019-12-21T15:45:05.465 INFO:teuthology.orchestra.run.smithi172.stdout:Installed: 2019-12-21T15:45:05.465 INFO:teuthology.orchestra.run.smithi172.stdout: rbd-fuse-2:15.0.0-8795.gc6e9f8f.el8.x86_64 2019-12-21T15:45:05.466 INFO:teuthology.orchestra.run.smithi172.stdout: 2019-12-21T15:45:05.466 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2019-12-21T15:45:05.631 INFO:teuthology.orchestra.run.smithi172:> sudo yum -y install rbd-mirror 2019-12-21T15:45:05.979 INFO:teuthology.orchestra.run.smithi172.stdout:Updating Subscription Management repositories. 2019-12-21T15:45:06.526 INFO:teuthology.orchestra.run.smithi172.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 88 kB/s | 2.8 kB 00:00 2019-12-21T15:45:06.585 INFO:teuthology.orchestra.run.smithi172.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 79 kB/s | 2.3 kB 00:00 2019-12-21T15:45:07.089 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2019-12-21T15:45:07.090 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2019-12-21T15:45:07.091 INFO:teuthology.orchestra.run.smithi172.stdout: Package Arch Version Repository 2019-12-21T15:45:07.091 INFO:teuthology.orchestra.run.smithi172.stdout: Size 2019-12-21T15:45:07.091 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2019-12-21T15:45:07.091 INFO:teuthology.orchestra.run.smithi172.stdout:Installing: 2019-12-21T15:45:07.091 INFO:teuthology.orchestra.run.smithi172.stdout: rbd-mirror x86_64 2:15.0.0-8795.gc6e9f8f.el8 ceph 2.5 M 2019-12-21T15:45:07.091 INFO:teuthology.orchestra.run.smithi172.stdout: 2019-12-21T15:45:07.092 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction Summary 2019-12-21T15:45:07.092 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2019-12-21T15:45:07.092 INFO:teuthology.orchestra.run.smithi172.stdout:Install 1 Package 2019-12-21T15:45:07.092 INFO:teuthology.orchestra.run.smithi172.stdout: 2019-12-21T15:45:07.092 INFO:teuthology.orchestra.run.smithi172.stdout:Total download size: 2.5 M 2019-12-21T15:45:07.092 INFO:teuthology.orchestra.run.smithi172.stdout:Installed size: 11 M 2019-12-21T15:45:07.093 INFO:teuthology.orchestra.run.smithi172.stdout:Downloading Packages: 2019-12-21T15:45:07.358 INFO:teuthology.orchestra.run.smithi172.stdout:rbd-mirror-15.0.0-8795.gc6e9f8f.el8.x86_64.rpm 9.5 MB/s | 2.5 MB 00:00 2019-12-21T15:45:07.359 INFO:teuthology.orchestra.run.smithi172.stdout:-------------------------------------------------------------------------------- 2019-12-21T15:45:07.359 INFO:teuthology.orchestra.run.smithi172.stdout:Total 9.5 MB/s | 2.5 MB 00:00 2019-12-21T15:45:07.359 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction check 2019-12-21T15:45:07.371 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction check succeeded. 2019-12-21T15:45:07.371 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction test 2019-12-21T15:45:07.429 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction test succeeded. 2019-12-21T15:45:07.429 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction 2019-12-21T15:45:07.789 INFO:teuthology.orchestra.run.smithi172.stdout: Preparing : 1/1 2019-12-21T15:45:07.980 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : rbd-mirror-2:15.0.0-8795.gc6e9f8f.el8.x86_64 1/1 2019-12-21T15:45:08.282 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: rbd-mirror-2:15.0.0-8795.gc6e9f8f.el8.x86_64 1/1 2019-12-21T15:45:08.623 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : rbd-mirror-2:15.0.0-8795.gc6e9f8f.el8.x86_64 1/1 2019-12-21T15:45:08.623 INFO:teuthology.orchestra.run.smithi172.stdout:Installed products updated. 2019-12-21T15:45:10.111 INFO:teuthology.orchestra.run.smithi172.stdout: 2019-12-21T15:45:10.112 INFO:teuthology.orchestra.run.smithi172.stdout:Installed: 2019-12-21T15:45:10.112 INFO:teuthology.orchestra.run.smithi172.stdout: rbd-mirror-2:15.0.0-8795.gc6e9f8f.el8.x86_64 2019-12-21T15:45:10.112 INFO:teuthology.orchestra.run.smithi172.stdout: 2019-12-21T15:45:10.112 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2019-12-21T15:45:10.265 INFO:teuthology.orchestra.run.smithi172:> sudo yum -y install rbd-nbd 2019-12-21T15:45:10.614 INFO:teuthology.orchestra.run.smithi172.stdout:Updating Subscription Management repositories. 2019-12-21T15:45:11.171 INFO:teuthology.orchestra.run.smithi172.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 83 kB/s | 2.8 kB 00:00 2019-12-21T15:45:11.234 INFO:teuthology.orchestra.run.smithi172.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 71 kB/s | 2.3 kB 00:00 2019-12-21T15:45:11.750 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2019-12-21T15:45:11.751 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2019-12-21T15:45:11.751 INFO:teuthology.orchestra.run.smithi172.stdout: Package Arch Version Repository Size 2019-12-21T15:45:11.751 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2019-12-21T15:45:11.751 INFO:teuthology.orchestra.run.smithi172.stdout:Installing: 2019-12-21T15:45:11.751 INFO:teuthology.orchestra.run.smithi172.stdout: rbd-nbd x86_64 2:15.0.0-8795.gc6e9f8f.el8 ceph 140 k 2019-12-21T15:45:11.752 INFO:teuthology.orchestra.run.smithi172.stdout: 2019-12-21T15:45:11.752 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction Summary 2019-12-21T15:45:11.752 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2019-12-21T15:45:11.752 INFO:teuthology.orchestra.run.smithi172.stdout:Install 1 Package 2019-12-21T15:45:11.752 INFO:teuthology.orchestra.run.smithi172.stdout: 2019-12-21T15:45:11.752 INFO:teuthology.orchestra.run.smithi172.stdout:Total download size: 140 k 2019-12-21T15:45:11.753 INFO:teuthology.orchestra.run.smithi172.stdout:Installed size: 504 k 2019-12-21T15:45:11.753 INFO:teuthology.orchestra.run.smithi172.stdout:Downloading Packages: 2019-12-21T15:45:11.903 INFO:teuthology.orchestra.run.smithi172.stdout:rbd-nbd-15.0.0-8795.gc6e9f8f.el8.x86_64.rpm 924 kB/s | 140 kB 00:00 2019-12-21T15:45:11.904 INFO:teuthology.orchestra.run.smithi172.stdout:-------------------------------------------------------------------------------- 2019-12-21T15:45:11.904 INFO:teuthology.orchestra.run.smithi172.stdout:Total 917 kB/s | 140 kB 00:00 2019-12-21T15:45:11.904 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction check 2019-12-21T15:45:11.914 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction check succeeded. 2019-12-21T15:45:11.914 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction test 2019-12-21T15:45:11.962 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction test succeeded. 2019-12-21T15:45:11.963 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction 2019-12-21T15:45:12.066 INFO:teuthology.orchestra.run.smithi172.stdout: Preparing : 1/1 2019-12-21T15:45:12.260 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : rbd-nbd-2:15.0.0-8795.gc6e9f8f.el8.x86_64 1/1 2019-12-21T15:45:12.459 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: rbd-nbd-2:15.0.0-8795.gc6e9f8f.el8.x86_64 1/1 2019-12-21T15:45:12.791 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : rbd-nbd-2:15.0.0-8795.gc6e9f8f.el8.x86_64 1/1 2019-12-21T15:45:12.792 INFO:teuthology.orchestra.run.smithi172.stdout:Installed products updated. 2019-12-21T15:45:14.412 INFO:teuthology.orchestra.run.smithi172.stdout: 2019-12-21T15:45:14.412 INFO:teuthology.orchestra.run.smithi172.stdout:Installed: 2019-12-21T15:45:14.412 INFO:teuthology.orchestra.run.smithi172.stdout: rbd-nbd-2:15.0.0-8795.gc6e9f8f.el8.x86_64 2019-12-21T15:45:14.413 INFO:teuthology.orchestra.run.smithi172.stdout: 2019-12-21T15:45:14.413 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2019-12-21T15:45:14.562 DEBUG:teuthology.parallel:result is None 2019-12-21T15:45:14.562 INFO:teuthology.orchestra.run.smithi172:> sudo lsb_release -is 2019-12-21T15:45:14.600 INFO:teuthology.orchestra.run.smithi172.stdout:RedHatEnterprise 2019-12-21T15:45:14.601 DEBUG:teuthology.misc:System to be installed: RedHatEnterprise 2019-12-21T15:45:14.601 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=c6e9f8f9e7516809792ffb509ef4a897f5c2f45e 2019-12-21T15:45:14.763 INFO:teuthology.orchestra.run.smithi172:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2019-12-21T15:45:14.781 INFO:teuthology.orchestra.run.smithi172.stdout:15.0.0-8795.gc6e9f8f.el8 2019-12-21T15:45:14.782 INFO:teuthology.packaging:The installed version of ceph is 15.0.0-8795.gc6e9f8f.el8 2019-12-21T15:45:14.782 INFO:teuthology.task.install:The correct ceph version 15.0.0-8795.gc6e9f8f is installed. 2019-12-21T15:45:14.782 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2019-12-21T15:45:14.783 INFO:teuthology.orchestra.run.smithi172:> sudo sh -c 'cat > /home/ubuntu/cephtest/valgrind.supp' 2019-12-21T15:45:14.849 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2019-12-21T15:45:14.850 INFO:teuthology.orchestra.run.smithi172:> sudo sh -c 'cat > /usr/bin/daemon-helper' 2019-12-21T15:45:14.920 INFO:teuthology.orchestra.run.smithi172:> sudo chmod a=rx -- /usr/bin/daemon-helper 2019-12-21T15:45:14.986 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2019-12-21T15:45:14.986 INFO:teuthology.orchestra.run.smithi172:> sudo sh -c 'cat > /usr/bin/adjust-ulimits' 2019-12-21T15:45:15.072 INFO:teuthology.orchestra.run.smithi172:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2019-12-21T15:45:15.138 INFO:teuthology.run_tasks:Running task workunit... 2019-12-21T15:45:15.156 INFO:tasks.workunit:Pulling workunits from ref 2b368c3feb5f798e3a142f2a972094a812d3d958 2019-12-21T15:45:15.157 INFO:tasks.workunit:Making a separate scratch dir for every client... 2019-12-21T15:45:15.157 INFO:tasks.workunit:timeout=3h 2019-12-21T15:45:15.157 INFO:tasks.workunit:cleanup=True 2019-12-21T15:45:15.157 INFO:teuthology.orchestra.run.smithi172:> stat -- /home/ubuntu/cephtest/mnt.0 2019-12-21T15:45:15.193 INFO:teuthology.orchestra.run.smithi172.stderr:stat: cannot stat '/home/ubuntu/cephtest/mnt.0': No such file or directory 2019-12-21T15:45:15.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2019-12-21T15:45:15.193 INFO:teuthology.orchestra.run.smithi172:> mkdir -- /home/ubuntu/cephtest/mnt.0 2019-12-21T15:45:15.246 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2019-12-21T15:45:15.247 INFO:teuthology.orchestra.run.smithi172:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2019-12-21T15:45:15.301 INFO:teuthology.orchestra.run.smithi172:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone git://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 2b368c3feb5f798e3a142f2a972094a812d3d958 2019-12-21T15:45:15.409 INFO:tasks.workunit.client.0.smithi172.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2019-12-21T15:46:21.330 INFO:tasks.workunit.client.0.smithi172.stderr:Note: checking out '2b368c3feb5f798e3a142f2a972094a812d3d958'. 2019-12-21T15:46:21.330 INFO:tasks.workunit.client.0.smithi172.stderr: 2019-12-21T15:46:21.330 INFO:tasks.workunit.client.0.smithi172.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2019-12-21T15:46:21.330 INFO:tasks.workunit.client.0.smithi172.stderr:changes and commit them, and you can discard any commits you make in this 2019-12-21T15:46:21.331 INFO:tasks.workunit.client.0.smithi172.stderr:state without impacting any branches by performing another checkout. 2019-12-21T15:46:21.331 INFO:tasks.workunit.client.0.smithi172.stderr: 2019-12-21T15:46:21.331 INFO:tasks.workunit.client.0.smithi172.stderr:If you want to create a new branch to retain commits you create, you may 2019-12-21T15:46:21.331 INFO:tasks.workunit.client.0.smithi172.stderr:do so (now or later) by using -b with the checkout command again. Example: 2019-12-21T15:46:21.331 INFO:tasks.workunit.client.0.smithi172.stderr: 2019-12-21T15:46:21.331 INFO:tasks.workunit.client.0.smithi172.stderr: git checkout -b 2019-12-21T15:46:21.331 INFO:tasks.workunit.client.0.smithi172.stderr: 2019-12-21T15:46:21.332 INFO:tasks.workunit.client.0.smithi172.stderr:HEAD is now at 2b368c3feb Merge branch 'wip-old-clients-cephadm' of https://github.com/liewegas/ceph into wip-sage-testing-2019-12-21-0902 2019-12-21T15:46:21.334 INFO:teuthology.orchestra.run.smithi172:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/standalone && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2019-12-21T15:46:21.405 DEBUG:teuthology.orchestra.remote:smithi172:/home/ubuntu/cephtest/workunits.list.client.0 is 1KB 2019-12-21T15:46:21.452 INFO:tasks.workunit:Running workunits matching scrub on client.0... 2019-12-21T15:46:21.452 INFO:tasks.workunit:Running workunit scrub/osd-recovery-scrub.sh... 2019-12-21T15:46:21.453 INFO:teuthology.orchestra.run.smithi172:workunit test scrub/osd-recovery-scrub.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=2b368c3feb5f798e3a142f2a972094a812d3d958 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh 2019-12-21T15:46:21.500 INFO:tasks.workunit.client.0.smithi172.stderr:stty: 'standard input': Inappropriate ioctl for device 2019-12-21T15:46:21.507 INFO:tasks.workunit.client.0.smithi172.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2019-12-21T15:46:21.507 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2050: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2019-12-21T15:46:21.507 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2050: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2019-12-21T15:46:21.508 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2051: main: export PYTHONWARNINGS=ignore 2019-12-21T15:46:21.508 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2051: main: PYTHONWARNINGS=ignore 2019-12-21T15:46:21.508 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2052: main: export CEPH_CONF=/dev/null 2019-12-21T15:46:21.508 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2052: main: CEPH_CONF=/dev/null 2019-12-21T15:46:21.508 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2053: main: unset CEPH_ARGS 2019-12-21T15:46:21.508 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2055: main: local code 2019-12-21T15:46:21.508 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2056: main: run td/osd-recovery-scrub 2019-12-21T15:46:21.509 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:20: run: local dir=td/osd-recovery-scrub 2019-12-21T15:46:21.509 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:21: run: shift 2019-12-21T15:46:21.509 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:23: run: export CEPH_MON=127.0.0.1:7124 2019-12-21T15:46:21.509 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:23: run: CEPH_MON=127.0.0.1:7124 2019-12-21T15:46:21.509 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:24: run: export CEPH_ARGS 2019-12-21T15:46:21.509 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:25: run: uuidgen 2019-12-21T15:46:21.517 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:25: run: CEPH_ARGS+='--fsid=72defa8a-f350-48c4-bd40-c303a855171e --auth-supported=none ' 2019-12-21T15:46:21.517 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:26: run: CEPH_ARGS+='--mon-host=127.0.0.1:7124 ' 2019-12-21T15:46:21.517 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:28: run: export -n CEPH_CLI_TEST_DUP_COMMAND 2019-12-21T15:46:21.517 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:29: run: set 2019-12-21T15:46:21.517 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:29: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2019-12-21T15:46:21.520 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:29: run: local funcs=TEST_recovery_scrub 2019-12-21T15:46:21.520 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:30: run: for func in $funcs 2019-12-21T15:46:21.520 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:31: run: TEST_recovery_scrub td/osd-recovery-scrub 2019-12-21T15:46:21.520 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:36: TEST_recovery_scrub: local dir=td/osd-recovery-scrub 2019-12-21T15:46:21.520 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:37: TEST_recovery_scrub: local poolname=test 2019-12-21T15:46:21.520 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:39: TEST_recovery_scrub: TESTDATA=testdata.25813 2019-12-21T15:46:21.521 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:40: TEST_recovery_scrub: OSDS=8 2019-12-21T15:46:21.521 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:41: TEST_recovery_scrub: PGS=32 2019-12-21T15:46:21.521 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:42: TEST_recovery_scrub: OBJECTS=4 2019-12-21T15:46:21.521 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:44: TEST_recovery_scrub: setup td/osd-recovery-scrub 2019-12-21T15:46:21.521 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-recovery-scrub 2019-12-21T15:46:21.521 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-recovery-scrub 2019-12-21T15:46:21.521 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-scrub 2019-12-21T15:46:21.522 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2019-12-21T15:46:21.522 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-scrub KILL 2019-12-21T15:46:21.522 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2019-12-21T15:46:21.522 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2019-12-21T15:46:21.522 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2019-12-21T15:46:21.522 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2019-12-21T15:46:21.522 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2019-12-21T15:46:21.524 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2019-12-21T15:46:21.524 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2019-12-21T15:46:21.524 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2019-12-21T15:46:21.525 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2019-12-21T15:46:21.525 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2019-12-21T15:46:21.525 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2019-12-21T15:46:21.525 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2019-12-21T15:46:21.525 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local 'pattern=|/usr/lib/systemd/systemd-coredump %P %u %g %s %t %c %h %e' 2019-12-21T15:46:21.525 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' '|' = '|' ']' 2019-12-21T15:46:21.525 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:177: teardown: pattern= 2019-12-21T15:46:21.526 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2019-12-21T15:46:21.526 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname '' 2019-12-21T15:46:21.527 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls . 2019-12-21T15:46:21.527 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2019-12-21T15:46:21.527 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-scrub 2019-12-21T15:46:21.527 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2019-12-21T15:46:21.527 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:46:21.527 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.25813 2019-12-21T15:46:21.528 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.25813 2019-12-21T15:46:21.528 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2019-12-21T15:46:21.529 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2019-12-21T15:46:21.529 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-recovery-scrub 2019-12-21T15:46:21.529 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2019-12-21T15:46:21.529 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:46:21.529 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.25813 2019-12-21T15:46:21.529 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.25813 2019-12-21T15:46:21.531 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2019-12-21T15:46:21.531 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2019-12-21T15:46:21.531 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2019-12-21T15:46:21.531 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2019-12-21T15:46:21.531 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-recovery-scrub 1' TERM HUP INT 2019-12-21T15:46:21.532 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:45: TEST_recovery_scrub: run_mon td/osd-recovery-scrub a --osd_pool_default_size=1 2019-12-21T15:46:21.532 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-recovery-scrub 2019-12-21T15:46:21.532 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2019-12-21T15:46:21.532 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2019-12-21T15:46:21.532 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2019-12-21T15:46:21.532 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-recovery-scrub/a 2019-12-21T15:46:21.532 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-recovery-scrub/a --run-dir=td/osd-recovery-scrub --osd_pool_default_size=1 2019-12-21T15:46:21.802 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2019-12-21T15:46:21.802 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2019-12-21T15:46:21.802 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2019-12-21T15:46:21.802 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2019-12-21T15:46:21.802 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:46:21.802 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.25813 2019-12-21T15:46:21.803 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.25813/$cluster-$name.asok' 2019-12-21T15:46:21.803 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-recovery-scrub/a '--log-file=td/osd-recovery-scrub/$name.log' '--admin-socket=/tmp/ceph-asok.25813/$cluster-$name.asok' --mon-cluster-log-file=td/osd-recovery-scrub/log --run-dir=td/osd-recovery-scrub '--pid-file=td/osd-recovery-scrub/$name.pid' --mon-allow-pool-delete --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --osd_pool_default_size=1 2019-12-21T15:46:21.955 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: cat 2019-12-21T15:46:21.955 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: get_config mon a fsid 2019-12-21T15:46:21.955 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: get_config: local daemon=mon 2019-12-21T15:46:21.955 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: get_config: local id=a 2019-12-21T15:46:21.955 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local config=fsid 2019-12-21T15:46:21.956 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: jq -r .fsid 2019-12-21T15:46:21.956 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: get_asok_path mon.a 2019-12-21T15:46:21.956 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2019-12-21T15:46:21.956 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2019-12-21T15:46:21.956 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:46:21.956 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:46:21.957 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.25813 2019-12-21T15:46:21.957 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.25813/ceph-mon.a.asok 2019-12-21T15:46:21.957 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: CEPH_ARGS= 2019-12-21T15:46:21.957 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: ceph --format json daemon /tmp/ceph-asok.25813/ceph-mon.a.asok config get fsid 2019-12-21T15:46:22.042 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: get_config mon a mon_host 2019-12-21T15:46:22.042 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: get_config: local daemon=mon 2019-12-21T15:46:22.042 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: get_config: local id=a 2019-12-21T15:46:22.043 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local config=mon_host 2019-12-21T15:46:22.043 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: jq -r .mon_host 2019-12-21T15:46:22.043 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: get_asok_path mon.a 2019-12-21T15:46:22.043 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2019-12-21T15:46:22.043 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2019-12-21T15:46:22.043 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:46:22.043 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:46:22.044 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.25813 2019-12-21T15:46:22.044 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.25813/ceph-mon.a.asok 2019-12-21T15:46:22.044 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: CEPH_ARGS= 2019-12-21T15:46:22.044 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: ceph --format json daemon /tmp/ceph-asok.25813/ceph-mon.a.asok config get mon_host 2019-12-21T15:46:22.129 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:46: TEST_recovery_scrub: run_mgr td/osd-recovery-scrub x 2019-12-21T15:46:22.130 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:551: run_mgr: local dir=td/osd-recovery-scrub 2019-12-21T15:46:22.130 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: shift 2019-12-21T15:46:22.130 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local id=x 2019-12-21T15:46:22.130 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2019-12-21T15:46:22.130 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local data=td/osd-recovery-scrub/x 2019-12-21T15:46:22.131 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:558: run_mgr: get_asok_path 2019-12-21T15:46:22.131 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2019-12-21T15:46:22.131 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2019-12-21T15:46:22.131 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2019-12-21T15:46:22.131 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:46:22.131 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.25813 2019-12-21T15:46:22.131 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.25813/$cluster-$name.asok' 2019-12-21T15:46:22.132 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:558: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2019-12-21T15:46:22.132 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:558: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-recovery-scrub/x '--log-file=td/osd-recovery-scrub/$name.log' '--admin-socket=/tmp/ceph-asok.25813/$cluster-$name.asok' --run-dir=td/osd-recovery-scrub '--pid-file=td/osd-recovery-scrub/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2019-12-21T15:46:22.751 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:47: TEST_recovery_scrub: expr 8 - 1 2019-12-21T15:46:22.752 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:47: TEST_recovery_scrub: seq 0 7 2019-12-21T15:46:22.760 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:47: TEST_recovery_scrub: for osd in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:46:22.760 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:49: TEST_recovery_scrub: run_osd td/osd-recovery-scrub 0 2019-12-21T15:46:22.761 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local dir=td/osd-recovery-scrub 2019-12-21T15:46:22.761 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: shift 2019-12-21T15:46:22.761 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local id=0 2019-12-21T15:46:22.761 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2019-12-21T15:46:22.761 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local osd_data=td/osd-recovery-scrub/0 2019-12-21T15:46:22.761 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local 'ceph_args=--fsid=72defa8a-f350-48c4-bd40-c303a855171e --auth-supported=none --mon-host=127.0.0.1:7124 ' 2019-12-21T15:46:22.762 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2019-12-21T15:46:22.762 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-journal-size=100' 2019-12-21T15:46:22.762 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2019-12-21T15:46:22.762 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/0' 2019-12-21T15:46:22.762 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/0/journal' 2019-12-21T15:46:22.762 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --chdir=' 2019-12-21T15:46:22.763 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+= 2019-12-21T15:46:22.763 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2019-12-21T15:46:22.763 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: get_asok_path 2019-12-21T15:46:22.763 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2019-12-21T15:46:22.763 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2019-12-21T15:46:22.763 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2019-12-21T15:46:22.763 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:46:22.764 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.25813 2019-12-21T15:46:22.764 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.25813/$cluster-$name.asok' 2019-12-21T15:46:22.764 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.25813/$cluster-$name.asok' 2019-12-21T15:46:22.764 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --debug-osd=20' 2019-12-21T15:46:22.764 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-ms=1' 2019-12-21T15:46:22.764 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-monc=20' 2019-12-21T15:46:22.765 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2019-12-21T15:46:22.765 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2019-12-21T15:46:22.765 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2019-12-21T15:46:22.765 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2019-12-21T15:46:22.765 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2019-12-21T15:46:22.765 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' ' 2019-12-21T15:46:22.766 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+= 2019-12-21T15:46:22.766 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: mkdir -p td/osd-recovery-scrub/0 2019-12-21T15:46:22.766 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: uuidgen 2019-12-21T15:46:22.766 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: local uuid=12e71992-a10d-4c19-bf6b-5e72beb20961 2019-12-21T15:46:22.766 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: echo 'add osd0 12e71992-a10d-4c19-bf6b-5e72beb20961' 2019-12-21T15:46:22.766 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: ceph-authtool --gen-print-key 2019-12-21T15:46:22.767 INFO:tasks.workunit.client.0.smithi172.stdout:add osd0 12e71992-a10d-4c19-bf6b-5e72beb20961 2019-12-21T15:46:22.775 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: OSD_SECRET=AQBOPv5dU83wLRAAvLmmuYwSUKGIbrcXpOQfEA== 2019-12-21T15:46:22.775 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo '{"cephx_secret": "AQBOPv5dU83wLRAAvLmmuYwSUKGIbrcXpOQfEA=="}' 2019-12-21T15:46:22.776 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph osd new 12e71992-a10d-4c19-bf6b-5e72beb20961 -i td/osd-recovery-scrub/0/new.json 2019-12-21T15:46:23.002 INFO:tasks.workunit.client.0.smithi172.stdout:0 2019-12-21T15:46:23.012 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: rm td/osd-recovery-scrub/0/new.json 2019-12-21T15:46:23.013 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-osd -i 0 --fsid=72defa8a-f350-48c4-bd40-c303a855171e --auth-supported=none --mon-host=127.0.0.1:7124 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/0 --osd-journal=td/osd-recovery-scrub/0/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.25813/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBOPv5dU83wLRAAvLmmuYwSUKGIbrcXpOQfEA== --osd-uuid 12e71992-a10d-4c19-bf6b-5e72beb20961 2019-12-21T15:46:23.032 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:46:23.030+0000 7fcc500d0ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T15:46:23.044 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:46:23.041+0000 7fcc500d0ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T15:46:23.046 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:46:23.043+0000 7fcc500d0ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T15:46:23.046 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:46:23.043+0000 7fcc500d0ec0 -1 bluestore(td/osd-recovery-scrub/0/block) _read_bdev_label failed to open td/osd-recovery-scrub/0/block: (2) No such file or directory 2019-12-21T15:46:23.046 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:46:23.043+0000 7fcc500d0ec0 -1 bluestore(td/osd-recovery-scrub/0/block) _read_bdev_label failed to open td/osd-recovery-scrub/0/block: (2) No such file or directory 2019-12-21T15:46:23.047 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:46:23.043+0000 7fcc500d0ec0 -1 bluestore(td/osd-recovery-scrub/0/block) _read_bdev_label failed to open td/osd-recovery-scrub/0/block: (2) No such file or directory 2019-12-21T15:46:23.102 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:46:23.100+0000 7fcc500d0ec0 -1 bluestore(td/osd-recovery-scrub/0) _read_fsid unparsable uuid 2019-12-21T15:46:26.085 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: local key_fn=td/osd-recovery-scrub/0/keyring 2019-12-21T15:46:26.085 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: cat 2019-12-21T15:46:26.086 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: echo adding osd0 key to auth repository 2019-12-21T15:46:26.086 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph -i td/osd-recovery-scrub/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2019-12-21T15:46:26.086 INFO:tasks.workunit.client.0.smithi172.stdout:adding osd0 key to auth repository 2019-12-21T15:46:26.262 INFO:tasks.workunit.client.0.smithi172.stdout:start osd.0 2019-12-21T15:46:26.262 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo start osd.0 2019-12-21T15:46:26.263 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph-osd -i 0 --fsid=72defa8a-f350-48c4-bd40-c303a855171e --auth-supported=none --mon-host=127.0.0.1:7124 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/0 --osd-journal=td/osd-recovery-scrub/0/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.25813/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2019-12-21T15:46:26.263 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: jq '.flags_set[]' 2019-12-21T15:46:26.263 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: grep -q '"noup"' 2019-12-21T15:46:26.263 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph osd dump --format=json 2019-12-21T15:46:26.280 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:46:26.277+0000 7fd829077ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T15:46:26.282 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:46:26.279+0000 7fd829077ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T15:46:26.283 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:46:26.280+0000 7fd829077ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T15:46:26.431 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: wait_for_osd up 0 2019-12-21T15:46:26.431 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:914: wait_for_osd: local state=up 2019-12-21T15:46:26.431 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local id=0 2019-12-21T15:46:26.431 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: status=1 2019-12-21T15:46:26.431 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i=0 )) 2019-12-21T15:46:26.431 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i < 300 )) 2019-12-21T15:46:26.432 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: echo 0 2019-12-21T15:46:26.432 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: grep 'osd.0 up' 2019-12-21T15:46:26.432 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: ceph osd dump 2019-12-21T15:46:26.432 INFO:tasks.workunit.client.0.smithi172.stdout:0 2019-12-21T15:46:26.601 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: sleep 1 2019-12-21T15:46:26.798 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:46:26.796+0000 7fd829077ec0 -1 Falling back to public interface 2019-12-21T15:46:27.342 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:46:27.340+0000 7fd829077ec0 -1 osd.0 0 log_to_monitors {default=true} 2019-12-21T15:46:27.603 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i++ )) 2019-12-21T15:46:27.603 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i < 300 )) 2019-12-21T15:46:27.603 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: echo 1 2019-12-21T15:46:27.603 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: ceph osd dump 2019-12-21T15:46:27.603 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: grep 'osd.0 up' 2019-12-21T15:46:27.603 INFO:tasks.workunit.client.0.smithi172.stdout:1 2019-12-21T15:46:27.833 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: sleep 1 2019-12-21T15:46:28.834 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i++ )) 2019-12-21T15:46:28.834 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i < 300 )) 2019-12-21T15:46:28.835 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: echo 2 2019-12-21T15:46:28.835 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: ceph osd dump 2019-12-21T15:46:28.835 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: grep 'osd.0 up' 2019-12-21T15:46:28.835 INFO:tasks.workunit.client.0.smithi172.stdout:2 2019-12-21T15:46:29.066 INFO:tasks.workunit.client.0.smithi172.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/26162,v1:127.0.0.1:6801/26162] [v2:127.0.0.1:6802/26162,v1:127.0.0.1:6803/26162] exists,up 12e71992-a10d-4c19-bf6b-5e72beb20961 2019-12-21T15:46:29.066 INFO:tasks.workunit.client.0.smithi172.stdout:add osd1 2c978e24-1d6b-4f75-916d-5b1cb8371ded 2019-12-21T15:46:29.066 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=0 2019-12-21T15:46:29.066 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: break 2019-12-21T15:46:29.066 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: return 0 2019-12-21T15:46:29.067 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:47: TEST_recovery_scrub: for osd in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:46:29.067 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:49: TEST_recovery_scrub: run_osd td/osd-recovery-scrub 1 2019-12-21T15:46:29.067 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local dir=td/osd-recovery-scrub 2019-12-21T15:46:29.067 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: shift 2019-12-21T15:46:29.067 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local id=1 2019-12-21T15:46:29.067 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2019-12-21T15:46:29.067 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local osd_data=td/osd-recovery-scrub/1 2019-12-21T15:46:29.068 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local 'ceph_args=--fsid=72defa8a-f350-48c4-bd40-c303a855171e --auth-supported=none --mon-host=127.0.0.1:7124 ' 2019-12-21T15:46:29.068 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2019-12-21T15:46:29.068 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-journal-size=100' 2019-12-21T15:46:29.068 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2019-12-21T15:46:29.069 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/1' 2019-12-21T15:46:29.069 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/1/journal' 2019-12-21T15:46:29.069 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --chdir=' 2019-12-21T15:46:29.070 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+= 2019-12-21T15:46:29.071 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2019-12-21T15:46:29.071 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: get_asok_path 2019-12-21T15:46:29.071 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2019-12-21T15:46:29.071 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2019-12-21T15:46:29.071 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2019-12-21T15:46:29.071 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:46:29.071 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.25813 2019-12-21T15:46:29.072 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.25813/$cluster-$name.asok' 2019-12-21T15:46:29.072 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.25813/$cluster-$name.asok' 2019-12-21T15:46:29.072 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --debug-osd=20' 2019-12-21T15:46:29.072 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-ms=1' 2019-12-21T15:46:29.072 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-monc=20' 2019-12-21T15:46:29.072 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2019-12-21T15:46:29.072 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2019-12-21T15:46:29.073 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2019-12-21T15:46:29.073 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2019-12-21T15:46:29.073 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2019-12-21T15:46:29.073 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' ' 2019-12-21T15:46:29.073 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+= 2019-12-21T15:46:29.073 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: mkdir -p td/osd-recovery-scrub/1 2019-12-21T15:46:29.073 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: uuidgen 2019-12-21T15:46:29.074 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: local uuid=2c978e24-1d6b-4f75-916d-5b1cb8371ded 2019-12-21T15:46:29.074 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: echo 'add osd1 2c978e24-1d6b-4f75-916d-5b1cb8371ded' 2019-12-21T15:46:29.074 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: ceph-authtool --gen-print-key 2019-12-21T15:46:29.077 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: OSD_SECRET=AQBVPv5d/uZ6BBAAqZWqGNp3nMmAQHcwBGRXPg== 2019-12-21T15:46:29.077 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo '{"cephx_secret": "AQBVPv5d/uZ6BBAAqZWqGNp3nMmAQHcwBGRXPg=="}' 2019-12-21T15:46:29.077 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph osd new 2c978e24-1d6b-4f75-916d-5b1cb8371ded -i td/osd-recovery-scrub/1/new.json 2019-12-21T15:46:29.384 INFO:tasks.workunit.client.0.smithi172.stdout:1 2019-12-21T15:46:29.394 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: rm td/osd-recovery-scrub/1/new.json 2019-12-21T15:46:29.395 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-osd -i 1 --fsid=72defa8a-f350-48c4-bd40-c303a855171e --auth-supported=none --mon-host=127.0.0.1:7124 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/1 --osd-journal=td/osd-recovery-scrub/1/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.25813/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBVPv5d/uZ6BBAAqZWqGNp3nMmAQHcwBGRXPg== --osd-uuid 2c978e24-1d6b-4f75-916d-5b1cb8371ded 2019-12-21T15:46:29.414 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:46:29.411+0000 7f27d4ba3ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T15:46:29.417 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:46:29.414+0000 7f27d4ba3ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T15:46:29.418 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:46:29.416+0000 7f27d4ba3ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T15:46:29.419 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:46:29.416+0000 7f27d4ba3ec0 -1 bluestore(td/osd-recovery-scrub/1/block) _read_bdev_label failed to open td/osd-recovery-scrub/1/block: (2) No such file or directory 2019-12-21T15:46:29.419 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:46:29.416+0000 7f27d4ba3ec0 -1 bluestore(td/osd-recovery-scrub/1/block) _read_bdev_label failed to open td/osd-recovery-scrub/1/block: (2) No such file or directory 2019-12-21T15:46:29.419 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:46:29.416+0000 7f27d4ba3ec0 -1 bluestore(td/osd-recovery-scrub/1/block) _read_bdev_label failed to open td/osd-recovery-scrub/1/block: (2) No such file or directory 2019-12-21T15:46:29.485 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:46:29.482+0000 7f27d4ba3ec0 -1 bluestore(td/osd-recovery-scrub/1) _read_fsid unparsable uuid 2019-12-21T15:46:32.458 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: local key_fn=td/osd-recovery-scrub/1/keyring 2019-12-21T15:46:32.459 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: cat 2019-12-21T15:46:32.460 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: echo adding osd1 key to auth repository 2019-12-21T15:46:32.460 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph -i td/osd-recovery-scrub/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2019-12-21T15:46:32.460 INFO:tasks.workunit.client.0.smithi172.stdout:adding osd1 key to auth repository 2019-12-21T15:46:32.742 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo start osd.1 2019-12-21T15:46:32.742 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph-osd -i 1 --fsid=72defa8a-f350-48c4-bd40-c303a855171e --auth-supported=none --mon-host=127.0.0.1:7124 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/1 --osd-journal=td/osd-recovery-scrub/1/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.25813/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2019-12-21T15:46:32.742 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph osd dump --format=json 2019-12-21T15:46:32.742 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: grep -q '"noup"' 2019-12-21T15:46:32.742 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: jq '.flags_set[]' 2019-12-21T15:46:32.743 INFO:tasks.workunit.client.0.smithi172.stdout:start osd.1 2019-12-21T15:46:32.760 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:46:32.758+0000 7fe44d8b1ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T15:46:32.762 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:46:32.760+0000 7fe44d8b1ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T15:46:32.768 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:46:32.762+0000 7fe44d8b1ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T15:46:32.971 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: wait_for_osd up 1 2019-12-21T15:46:32.971 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:914: wait_for_osd: local state=up 2019-12-21T15:46:32.972 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local id=1 2019-12-21T15:46:32.972 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: status=1 2019-12-21T15:46:32.972 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i=0 )) 2019-12-21T15:46:32.972 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i < 300 )) 2019-12-21T15:46:32.972 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: echo 0 2019-12-21T15:46:32.972 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: ceph osd dump 2019-12-21T15:46:32.973 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: grep 'osd.1 up' 2019-12-21T15:46:32.973 INFO:tasks.workunit.client.0.smithi172.stdout:0 2019-12-21T15:46:33.195 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: sleep 1 2019-12-21T15:46:33.279 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:46:33.277+0000 7fe44d8b1ec0 -1 Falling back to public interface 2019-12-21T15:46:33.965 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:46:33.963+0000 7fe44d8b1ec0 -1 osd.1 0 log_to_monitors {default=true} 2019-12-21T15:46:34.197 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i++ )) 2019-12-21T15:46:34.197 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i < 300 )) 2019-12-21T15:46:34.197 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: echo 1 2019-12-21T15:46:34.197 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: ceph osd dump 2019-12-21T15:46:34.197 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: grep 'osd.1 up' 2019-12-21T15:46:34.198 INFO:tasks.workunit.client.0.smithi172.stdout:1 2019-12-21T15:46:34.463 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: sleep 1 2019-12-21T15:46:35.465 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i++ )) 2019-12-21T15:46:35.465 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i < 300 )) 2019-12-21T15:46:35.465 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: echo 2 2019-12-21T15:46:35.465 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: ceph osd dump 2019-12-21T15:46:35.466 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: grep 'osd.1 up' 2019-12-21T15:46:35.466 INFO:tasks.workunit.client.0.smithi172.stdout:2 2019-12-21T15:46:35.697 INFO:tasks.workunit.client.0.smithi172.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/26566,v1:127.0.0.1:6811/26566] [v2:127.0.0.1:6812/26566,v1:127.0.0.1:6813/26566] exists,up 2c978e24-1d6b-4f75-916d-5b1cb8371ded 2019-12-21T15:46:35.697 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=0 2019-12-21T15:46:35.698 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: break 2019-12-21T15:46:35.698 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: return 0 2019-12-21T15:46:35.698 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:47: TEST_recovery_scrub: for osd in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:46:35.698 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:49: TEST_recovery_scrub: run_osd td/osd-recovery-scrub 2 2019-12-21T15:46:35.698 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local dir=td/osd-recovery-scrub 2019-12-21T15:46:35.698 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: shift 2019-12-21T15:46:35.699 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local id=2 2019-12-21T15:46:35.699 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2019-12-21T15:46:35.699 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local osd_data=td/osd-recovery-scrub/2 2019-12-21T15:46:35.699 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local 'ceph_args=--fsid=72defa8a-f350-48c4-bd40-c303a855171e --auth-supported=none --mon-host=127.0.0.1:7124 ' 2019-12-21T15:46:35.699 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2019-12-21T15:46:35.699 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-journal-size=100' 2019-12-21T15:46:35.699 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2019-12-21T15:46:35.700 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/2' 2019-12-21T15:46:35.700 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/2/journal' 2019-12-21T15:46:35.700 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --chdir=' 2019-12-21T15:46:35.700 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+= 2019-12-21T15:46:35.700 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2019-12-21T15:46:35.700 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: get_asok_path 2019-12-21T15:46:35.700 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2019-12-21T15:46:35.701 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2019-12-21T15:46:35.701 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2019-12-21T15:46:35.701 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:46:35.701 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.25813 2019-12-21T15:46:35.701 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.25813/$cluster-$name.asok' 2019-12-21T15:46:35.701 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.25813/$cluster-$name.asok' 2019-12-21T15:46:35.701 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --debug-osd=20' 2019-12-21T15:46:35.702 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-ms=1' 2019-12-21T15:46:35.702 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-monc=20' 2019-12-21T15:46:35.702 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2019-12-21T15:46:35.702 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2019-12-21T15:46:35.702 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2019-12-21T15:46:35.702 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2019-12-21T15:46:35.702 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2019-12-21T15:46:35.703 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' ' 2019-12-21T15:46:35.703 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+= 2019-12-21T15:46:35.703 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: mkdir -p td/osd-recovery-scrub/2 2019-12-21T15:46:35.704 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: uuidgen 2019-12-21T15:46:35.704 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: local uuid=48fb48d7-49ac-4a6c-aa2a-9116ce79bed2 2019-12-21T15:46:35.704 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: echo 'add osd2 48fb48d7-49ac-4a6c-aa2a-9116ce79bed2' 2019-12-21T15:46:35.704 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: ceph-authtool --gen-print-key 2019-12-21T15:46:35.704 INFO:tasks.workunit.client.0.smithi172.stdout:add osd2 48fb48d7-49ac-4a6c-aa2a-9116ce79bed2 2019-12-21T15:46:35.710 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: OSD_SECRET=AQBbPv5dFpNEKhAAUAAexjJc9lsqPrbRGypRPQ== 2019-12-21T15:46:35.711 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo '{"cephx_secret": "AQBbPv5dFpNEKhAAUAAexjJc9lsqPrbRGypRPQ=="}' 2019-12-21T15:46:35.711 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph osd new 48fb48d7-49ac-4a6c-aa2a-9116ce79bed2 -i td/osd-recovery-scrub/2/new.json 2019-12-21T15:46:36.031 INFO:tasks.workunit.client.0.smithi172.stdout:2 2019-12-21T15:46:36.041 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: rm td/osd-recovery-scrub/2/new.json 2019-12-21T15:46:36.042 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-osd -i 2 --fsid=72defa8a-f350-48c4-bd40-c303a855171e --auth-supported=none --mon-host=127.0.0.1:7124 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/2 --osd-journal=td/osd-recovery-scrub/2/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.25813/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBbPv5dFpNEKhAAUAAexjJc9lsqPrbRGypRPQ== --osd-uuid 48fb48d7-49ac-4a6c-aa2a-9116ce79bed2 2019-12-21T15:46:36.065 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:46:36.059+0000 7f1d7d7e2ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T15:46:36.066 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:46:36.061+0000 7f1d7d7e2ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T15:46:36.066 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:46:36.062+0000 7f1d7d7e2ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T15:46:36.066 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:46:36.062+0000 7f1d7d7e2ec0 -1 bluestore(td/osd-recovery-scrub/2/block) _read_bdev_label failed to open td/osd-recovery-scrub/2/block: (2) No such file or directory 2019-12-21T15:46:36.066 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:46:36.062+0000 7f1d7d7e2ec0 -1 bluestore(td/osd-recovery-scrub/2/block) _read_bdev_label failed to open td/osd-recovery-scrub/2/block: (2) No such file or directory 2019-12-21T15:46:36.066 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:46:36.062+0000 7f1d7d7e2ec0 -1 bluestore(td/osd-recovery-scrub/2/block) _read_bdev_label failed to open td/osd-recovery-scrub/2/block: (2) No such file or directory 2019-12-21T15:46:36.114 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:46:36.112+0000 7f1d7d7e2ec0 -1 bluestore(td/osd-recovery-scrub/2) _read_fsid unparsable uuid 2019-12-21T15:46:39.048 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: local key_fn=td/osd-recovery-scrub/2/keyring 2019-12-21T15:46:39.049 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: cat 2019-12-21T15:46:39.050 INFO:tasks.workunit.client.0.smithi172.stdout:adding osd2 key to auth repository 2019-12-21T15:46:39.050 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: echo adding osd2 key to auth repository 2019-12-21T15:46:39.050 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph -i td/osd-recovery-scrub/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2019-12-21T15:46:39.331 INFO:tasks.workunit.client.0.smithi172.stdout:start osd.2 2019-12-21T15:46:39.331 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo start osd.2 2019-12-21T15:46:39.332 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph osd dump --format=json 2019-12-21T15:46:39.332 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: jq '.flags_set[]' 2019-12-21T15:46:39.332 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: grep -q '"noup"' 2019-12-21T15:46:39.332 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph-osd -i 2 --fsid=72defa8a-f350-48c4-bd40-c303a855171e --auth-supported=none --mon-host=127.0.0.1:7124 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/2 --osd-journal=td/osd-recovery-scrub/2/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.25813/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2019-12-21T15:46:39.349 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:46:39.347+0000 7fbee22c5ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T15:46:39.351 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:46:39.349+0000 7fbee22c5ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T15:46:39.351 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:46:39.350+0000 7fbee22c5ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T15:46:39.556 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: wait_for_osd up 2 2019-12-21T15:46:39.556 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:914: wait_for_osd: local state=up 2019-12-21T15:46:39.557 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local id=2 2019-12-21T15:46:39.557 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: status=1 2019-12-21T15:46:39.557 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i=0 )) 2019-12-21T15:46:39.557 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i < 300 )) 2019-12-21T15:46:39.557 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: echo 0 2019-12-21T15:46:39.557 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: ceph osd dump 2019-12-21T15:46:39.557 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: grep 'osd.2 up' 2019-12-21T15:46:39.558 INFO:tasks.workunit.client.0.smithi172.stdout:0 2019-12-21T15:46:39.780 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: sleep 1 2019-12-21T15:46:39.867 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:46:39.866+0000 7fbee22c5ec0 -1 Falling back to public interface 2019-12-21T15:46:40.589 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:46:40.588+0000 7fbee22c5ec0 -1 osd.2 0 log_to_monitors {default=true} 2019-12-21T15:46:40.782 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i++ )) 2019-12-21T15:46:40.782 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i < 300 )) 2019-12-21T15:46:40.782 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: echo 1 2019-12-21T15:46:40.782 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: ceph osd dump 2019-12-21T15:46:40.782 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: grep 'osd.2 up' 2019-12-21T15:46:40.783 INFO:tasks.workunit.client.0.smithi172.stdout:1 2019-12-21T15:46:41.067 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: sleep 1 2019-12-21T15:46:42.068 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i++ )) 2019-12-21T15:46:42.068 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i < 300 )) 2019-12-21T15:46:42.069 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: echo 2 2019-12-21T15:46:42.069 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: ceph osd dump 2019-12-21T15:46:42.069 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: grep 'osd.2 up' 2019-12-21T15:46:42.069 INFO:tasks.workunit.client.0.smithi172.stdout:2 2019-12-21T15:46:42.300 INFO:tasks.workunit.client.0.smithi172.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/26943,v1:127.0.0.1:6819/26943] [v2:127.0.0.1:6820/26943,v1:127.0.0.1:6821/26943] exists,up 48fb48d7-49ac-4a6c-aa2a-9116ce79bed2 2019-12-21T15:46:42.301 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=0 2019-12-21T15:46:42.301 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: break 2019-12-21T15:46:42.301 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: return 0 2019-12-21T15:46:42.301 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:47: TEST_recovery_scrub: for osd in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:46:42.301 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:49: TEST_recovery_scrub: run_osd td/osd-recovery-scrub 3 2019-12-21T15:46:42.302 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local dir=td/osd-recovery-scrub 2019-12-21T15:46:42.302 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: shift 2019-12-21T15:46:42.302 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local id=3 2019-12-21T15:46:42.302 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2019-12-21T15:46:42.302 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local osd_data=td/osd-recovery-scrub/3 2019-12-21T15:46:42.302 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local 'ceph_args=--fsid=72defa8a-f350-48c4-bd40-c303a855171e --auth-supported=none --mon-host=127.0.0.1:7124 ' 2019-12-21T15:46:42.302 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2019-12-21T15:46:42.303 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-journal-size=100' 2019-12-21T15:46:42.303 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2019-12-21T15:46:42.303 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/3' 2019-12-21T15:46:42.303 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/3/journal' 2019-12-21T15:46:42.303 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --chdir=' 2019-12-21T15:46:42.303 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+= 2019-12-21T15:46:42.303 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2019-12-21T15:46:42.304 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: get_asok_path 2019-12-21T15:46:42.304 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2019-12-21T15:46:42.304 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2019-12-21T15:46:42.304 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2019-12-21T15:46:42.304 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:46:42.304 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.25813 2019-12-21T15:46:42.304 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.25813/$cluster-$name.asok' 2019-12-21T15:46:42.305 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.25813/$cluster-$name.asok' 2019-12-21T15:46:42.305 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --debug-osd=20' 2019-12-21T15:46:42.305 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-ms=1' 2019-12-21T15:46:42.305 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-monc=20' 2019-12-21T15:46:42.305 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2019-12-21T15:46:42.305 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2019-12-21T15:46:42.305 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2019-12-21T15:46:42.306 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2019-12-21T15:46:42.306 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2019-12-21T15:46:42.306 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' ' 2019-12-21T15:46:42.306 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+= 2019-12-21T15:46:42.306 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: mkdir -p td/osd-recovery-scrub/3 2019-12-21T15:46:42.307 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: uuidgen 2019-12-21T15:46:42.307 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: local uuid=14ba1109-2940-4d45-adc0-8508f5e6d013 2019-12-21T15:46:42.307 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: echo 'add osd3 14ba1109-2940-4d45-adc0-8508f5e6d013' 2019-12-21T15:46:42.307 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: ceph-authtool --gen-print-key 2019-12-21T15:46:42.308 INFO:tasks.workunit.client.0.smithi172.stdout:add osd3 14ba1109-2940-4d45-adc0-8508f5e6d013 2019-12-21T15:46:42.314 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: OSD_SECRET=AQBiPv5d/xKlEhAA/Ln7S43PZByZvlXHO6zbxQ== 2019-12-21T15:46:42.314 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo '{"cephx_secret": "AQBiPv5d/xKlEhAA/Ln7S43PZByZvlXHO6zbxQ=="}' 2019-12-21T15:46:42.314 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph osd new 14ba1109-2940-4d45-adc0-8508f5e6d013 -i td/osd-recovery-scrub/3/new.json 2019-12-21T15:46:42.601 INFO:tasks.workunit.client.0.smithi172.stdout:3 2019-12-21T15:46:42.606 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: rm td/osd-recovery-scrub/3/new.json 2019-12-21T15:46:42.607 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-osd -i 3 --fsid=72defa8a-f350-48c4-bd40-c303a855171e --auth-supported=none --mon-host=127.0.0.1:7124 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/3 --osd-journal=td/osd-recovery-scrub/3/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.25813/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBiPv5d/xKlEhAA/Ln7S43PZByZvlXHO6zbxQ== --osd-uuid 14ba1109-2940-4d45-adc0-8508f5e6d013 2019-12-21T15:46:42.626 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:46:42.625+0000 7f8e9e424ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T15:46:42.629 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:46:42.628+0000 7f8e9e424ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T15:46:42.631 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:46:42.629+0000 7f8e9e424ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T15:46:42.632 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:46:42.630+0000 7f8e9e424ec0 -1 bluestore(td/osd-recovery-scrub/3/block) _read_bdev_label failed to open td/osd-recovery-scrub/3/block: (2) No such file or directory 2019-12-21T15:46:42.632 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:46:42.630+0000 7f8e9e424ec0 -1 bluestore(td/osd-recovery-scrub/3/block) _read_bdev_label failed to open td/osd-recovery-scrub/3/block: (2) No such file or directory 2019-12-21T15:46:42.632 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:46:42.630+0000 7f8e9e424ec0 -1 bluestore(td/osd-recovery-scrub/3/block) _read_bdev_label failed to open td/osd-recovery-scrub/3/block: (2) No such file or directory 2019-12-21T15:46:42.679 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:46:42.678+0000 7f8e9e424ec0 -1 bluestore(td/osd-recovery-scrub/3) _read_fsid unparsable uuid 2019-12-21T15:46:45.716 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: local key_fn=td/osd-recovery-scrub/3/keyring 2019-12-21T15:46:45.716 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: cat 2019-12-21T15:46:45.716 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: echo adding osd3 key to auth repository 2019-12-21T15:46:45.716 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph -i td/osd-recovery-scrub/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2019-12-21T15:46:45.717 INFO:tasks.workunit.client.0.smithi172.stdout:adding osd3 key to auth repository 2019-12-21T15:46:45.998 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo start osd.3 2019-12-21T15:46:45.999 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph-osd -i 3 --fsid=72defa8a-f350-48c4-bd40-c303a855171e --auth-supported=none --mon-host=127.0.0.1:7124 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/3 --osd-journal=td/osd-recovery-scrub/3/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.25813/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2019-12-21T15:46:45.999 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: jq '.flags_set[]' 2019-12-21T15:46:45.999 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: grep -q '"noup"' 2019-12-21T15:46:45.999 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph osd dump --format=json 2019-12-21T15:46:46.000 INFO:tasks.workunit.client.0.smithi172.stdout:start osd.3 2019-12-21T15:46:46.014 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:46:46.012+0000 7f1fce873ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T15:46:46.016 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:46:46.014+0000 7f1fce873ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T15:46:46.019 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:46:46.016+0000 7f1fce873ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T15:46:46.224 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: wait_for_osd up 3 2019-12-21T15:46:46.224 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:914: wait_for_osd: local state=up 2019-12-21T15:46:46.224 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local id=3 2019-12-21T15:46:46.224 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: status=1 2019-12-21T15:46:46.225 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i=0 )) 2019-12-21T15:46:46.225 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i < 300 )) 2019-12-21T15:46:46.225 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: echo 0 2019-12-21T15:46:46.225 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: ceph osd dump 2019-12-21T15:46:46.225 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: grep 'osd.3 up' 2019-12-21T15:46:46.225 INFO:tasks.workunit.client.0.smithi172.stdout:0 2019-12-21T15:46:46.448 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: sleep 1 2019-12-21T15:46:46.533 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:46:46.531+0000 7f1fce873ec0 -1 Falling back to public interface 2019-12-21T15:46:47.262 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:46:47.261+0000 7f1fce873ec0 -1 osd.3 0 log_to_monitors {default=true} 2019-12-21T15:46:47.449 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i++ )) 2019-12-21T15:46:47.449 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i < 300 )) 2019-12-21T15:46:47.449 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: echo 1 2019-12-21T15:46:47.449 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: grep 'osd.3 up' 2019-12-21T15:46:47.450 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: ceph osd dump 2019-12-21T15:46:47.450 INFO:tasks.workunit.client.0.smithi172.stdout:1 2019-12-21T15:46:47.691 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: sleep 1 2019-12-21T15:46:48.692 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i++ )) 2019-12-21T15:46:48.693 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i < 300 )) 2019-12-21T15:46:48.693 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: echo 2 2019-12-21T15:46:48.693 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: ceph osd dump 2019-12-21T15:46:48.693 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: grep 'osd.3 up' 2019-12-21T15:46:48.693 INFO:tasks.workunit.client.0.smithi172.stdout:2 2019-12-21T15:46:48.935 INFO:tasks.workunit.client.0.smithi172.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/27320,v1:127.0.0.1:6827/27320] [v2:127.0.0.1:6828/27320,v1:127.0.0.1:6829/27320] exists,up 14ba1109-2940-4d45-adc0-8508f5e6d013 2019-12-21T15:46:48.936 INFO:tasks.workunit.client.0.smithi172.stdout:add osd4 46fba76b-b50e-494c-a915-bd1d2398af4b 2019-12-21T15:46:48.937 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=0 2019-12-21T15:46:48.937 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: break 2019-12-21T15:46:48.937 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: return 0 2019-12-21T15:46:48.937 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:47: TEST_recovery_scrub: for osd in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:46:48.938 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:49: TEST_recovery_scrub: run_osd td/osd-recovery-scrub 4 2019-12-21T15:46:48.938 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local dir=td/osd-recovery-scrub 2019-12-21T15:46:48.938 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: shift 2019-12-21T15:46:48.938 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local id=4 2019-12-21T15:46:48.939 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2019-12-21T15:46:48.939 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local osd_data=td/osd-recovery-scrub/4 2019-12-21T15:46:48.940 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local 'ceph_args=--fsid=72defa8a-f350-48c4-bd40-c303a855171e --auth-supported=none --mon-host=127.0.0.1:7124 ' 2019-12-21T15:46:48.940 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2019-12-21T15:46:48.940 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-journal-size=100' 2019-12-21T15:46:48.940 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2019-12-21T15:46:48.941 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/4' 2019-12-21T15:46:48.941 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/4/journal' 2019-12-21T15:46:48.941 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --chdir=' 2019-12-21T15:46:48.941 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+= 2019-12-21T15:46:48.942 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2019-12-21T15:46:48.942 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: get_asok_path 2019-12-21T15:46:48.942 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2019-12-21T15:46:48.942 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2019-12-21T15:46:48.942 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2019-12-21T15:46:48.943 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:46:48.943 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.25813 2019-12-21T15:46:48.943 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.25813/$cluster-$name.asok' 2019-12-21T15:46:48.943 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.25813/$cluster-$name.asok' 2019-12-21T15:46:48.944 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --debug-osd=20' 2019-12-21T15:46:48.944 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-ms=1' 2019-12-21T15:46:48.944 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-monc=20' 2019-12-21T15:46:48.944 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2019-12-21T15:46:48.945 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2019-12-21T15:46:48.945 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2019-12-21T15:46:48.945 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2019-12-21T15:46:48.945 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2019-12-21T15:46:48.945 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' ' 2019-12-21T15:46:48.946 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+= 2019-12-21T15:46:48.946 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: mkdir -p td/osd-recovery-scrub/4 2019-12-21T15:46:48.946 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: uuidgen 2019-12-21T15:46:48.946 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: local uuid=46fba76b-b50e-494c-a915-bd1d2398af4b 2019-12-21T15:46:48.947 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: echo 'add osd4 46fba76b-b50e-494c-a915-bd1d2398af4b' 2019-12-21T15:46:48.947 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: ceph-authtool --gen-print-key 2019-12-21T15:46:48.948 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: OSD_SECRET=AQBoPv5dvIlLOBAAe3InFmDnFUT3+z/Auesw/A== 2019-12-21T15:46:48.948 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo '{"cephx_secret": "AQBoPv5dvIlLOBAAe3InFmDnFUT3+z/Auesw/A=="}' 2019-12-21T15:46:48.948 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph osd new 46fba76b-b50e-494c-a915-bd1d2398af4b -i td/osd-recovery-scrub/4/new.json 2019-12-21T15:46:49.230 INFO:tasks.workunit.client.0.smithi172.stdout:4 2019-12-21T15:46:49.239 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: rm td/osd-recovery-scrub/4/new.json 2019-12-21T15:46:49.240 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-osd -i 4 --fsid=72defa8a-f350-48c4-bd40-c303a855171e --auth-supported=none --mon-host=127.0.0.1:7124 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/4 --osd-journal=td/osd-recovery-scrub/4/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.25813/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBoPv5dvIlLOBAAe3InFmDnFUT3+z/Auesw/A== --osd-uuid 46fba76b-b50e-494c-a915-bd1d2398af4b 2019-12-21T15:46:49.260 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:46:49.258+0000 7f9926f36ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T15:46:49.283 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:46:49.278+0000 7f9926f36ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T15:46:49.284 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:46:49.283+0000 7f9926f36ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T15:46:49.285 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:46:49.283+0000 7f9926f36ec0 -1 bluestore(td/osd-recovery-scrub/4/block) _read_bdev_label failed to open td/osd-recovery-scrub/4/block: (2) No such file or directory 2019-12-21T15:46:49.285 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:46:49.283+0000 7f9926f36ec0 -1 bluestore(td/osd-recovery-scrub/4/block) _read_bdev_label failed to open td/osd-recovery-scrub/4/block: (2) No such file or directory 2019-12-21T15:46:49.285 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:46:49.283+0000 7f9926f36ec0 -1 bluestore(td/osd-recovery-scrub/4/block) _read_bdev_label failed to open td/osd-recovery-scrub/4/block: (2) No such file or directory 2019-12-21T15:46:49.355 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:46:49.353+0000 7f9926f36ec0 -1 bluestore(td/osd-recovery-scrub/4) _read_fsid unparsable uuid 2019-12-21T15:46:52.417 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: local key_fn=td/osd-recovery-scrub/4/keyring 2019-12-21T15:46:52.417 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: cat 2019-12-21T15:46:52.418 INFO:tasks.workunit.client.0.smithi172.stdout:adding osd4 key to auth repository 2019-12-21T15:46:52.419 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: echo adding osd4 key to auth repository 2019-12-21T15:46:52.419 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph -i td/osd-recovery-scrub/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2019-12-21T15:46:52.707 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo start osd.4 2019-12-21T15:46:52.707 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph osd dump --format=json 2019-12-21T15:46:52.707 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph-osd -i 4 --fsid=72defa8a-f350-48c4-bd40-c303a855171e --auth-supported=none --mon-host=127.0.0.1:7124 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/4 --osd-journal=td/osd-recovery-scrub/4/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.25813/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2019-12-21T15:46:52.707 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: jq '.flags_set[]' 2019-12-21T15:46:52.708 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: grep -q '"noup"' 2019-12-21T15:46:52.708 INFO:tasks.workunit.client.0.smithi172.stdout:start osd.4 2019-12-21T15:46:52.725 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:46:52.722+0000 7feec2c9aec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T15:46:52.729 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:46:52.724+0000 7feec2c9aec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T15:46:52.729 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:46:52.725+0000 7feec2c9aec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T15:46:53.243 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:46:53.241+0000 7feec2c9aec0 -1 Falling back to public interface 2019-12-21T15:46:53.665 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: wait_for_osd up 4 2019-12-21T15:46:53.666 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:914: wait_for_osd: local state=up 2019-12-21T15:46:53.666 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local id=4 2019-12-21T15:46:53.666 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: status=1 2019-12-21T15:46:53.666 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i=0 )) 2019-12-21T15:46:53.667 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i < 300 )) 2019-12-21T15:46:53.667 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: echo 0 2019-12-21T15:46:53.667 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: grep 'osd.4 up' 2019-12-21T15:46:53.667 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: ceph osd dump 2019-12-21T15:46:53.668 INFO:tasks.workunit.client.0.smithi172.stdout:0 2019-12-21T15:46:53.903 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: sleep 1 2019-12-21T15:46:54.344 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:46:54.342+0000 7feec2c9aec0 -1 osd.4 0 log_to_monitors {default=true} 2019-12-21T15:46:54.904 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i++ )) 2019-12-21T15:46:54.904 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i < 300 )) 2019-12-21T15:46:54.905 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: echo 1 2019-12-21T15:46:54.905 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: ceph osd dump 2019-12-21T15:46:54.905 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: grep 'osd.4 up' 2019-12-21T15:46:54.905 INFO:tasks.workunit.client.0.smithi172.stdout:1 2019-12-21T15:46:55.147 INFO:tasks.workunit.client.0.smithi172.stdout:osd.4 up in weight 1 up_from 25 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/27697,v1:127.0.0.1:6835/27697] [v2:127.0.0.1:6836/27697,v1:127.0.0.1:6837/27697] exists,up 46fba76b-b50e-494c-a915-bd1d2398af4b 2019-12-21T15:46:55.148 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=0 2019-12-21T15:46:55.148 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: break 2019-12-21T15:46:55.148 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: return 0 2019-12-21T15:46:55.148 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:47: TEST_recovery_scrub: for osd in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:46:55.148 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:49: TEST_recovery_scrub: run_osd td/osd-recovery-scrub 5 2019-12-21T15:46:55.149 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local dir=td/osd-recovery-scrub 2019-12-21T15:46:55.149 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: shift 2019-12-21T15:46:55.149 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local id=5 2019-12-21T15:46:55.149 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2019-12-21T15:46:55.149 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local osd_data=td/osd-recovery-scrub/5 2019-12-21T15:46:55.149 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local 'ceph_args=--fsid=72defa8a-f350-48c4-bd40-c303a855171e --auth-supported=none --mon-host=127.0.0.1:7124 ' 2019-12-21T15:46:55.149 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2019-12-21T15:46:55.150 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-journal-size=100' 2019-12-21T15:46:55.150 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2019-12-21T15:46:55.150 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/5' 2019-12-21T15:46:55.150 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/5/journal' 2019-12-21T15:46:55.150 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --chdir=' 2019-12-21T15:46:55.150 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+= 2019-12-21T15:46:55.150 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2019-12-21T15:46:55.151 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: get_asok_path 2019-12-21T15:46:55.151 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2019-12-21T15:46:55.151 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2019-12-21T15:46:55.151 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2019-12-21T15:46:55.151 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:46:55.151 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.25813 2019-12-21T15:46:55.151 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.25813/$cluster-$name.asok' 2019-12-21T15:46:55.152 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.25813/$cluster-$name.asok' 2019-12-21T15:46:55.152 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --debug-osd=20' 2019-12-21T15:46:55.152 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-ms=1' 2019-12-21T15:46:55.152 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-monc=20' 2019-12-21T15:46:55.152 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2019-12-21T15:46:55.152 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2019-12-21T15:46:55.152 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2019-12-21T15:46:55.153 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2019-12-21T15:46:55.153 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2019-12-21T15:46:55.153 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' ' 2019-12-21T15:46:55.153 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+= 2019-12-21T15:46:55.153 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: mkdir -p td/osd-recovery-scrub/5 2019-12-21T15:46:55.154 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: uuidgen 2019-12-21T15:46:55.154 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: local uuid=7b76ca60-1092-4936-993f-4f50554de6dd 2019-12-21T15:46:55.154 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: echo 'add osd5 7b76ca60-1092-4936-993f-4f50554de6dd' 2019-12-21T15:46:55.155 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: ceph-authtool --gen-print-key 2019-12-21T15:46:55.155 INFO:tasks.workunit.client.0.smithi172.stdout:add osd5 7b76ca60-1092-4936-993f-4f50554de6dd 2019-12-21T15:46:55.161 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: OSD_SECRET=AQBvPv5dpBmECRAAvESm37WkFJ1GCk3TLzYrrg== 2019-12-21T15:46:55.161 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo '{"cephx_secret": "AQBvPv5dpBmECRAAvESm37WkFJ1GCk3TLzYrrg=="}' 2019-12-21T15:46:55.161 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph osd new 7b76ca60-1092-4936-993f-4f50554de6dd -i td/osd-recovery-scrub/5/new.json 2019-12-21T15:46:55.471 INFO:tasks.workunit.client.0.smithi172.stdout:5 2019-12-21T15:46:55.481 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: rm td/osd-recovery-scrub/5/new.json 2019-12-21T15:46:55.482 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-osd -i 5 --fsid=72defa8a-f350-48c4-bd40-c303a855171e --auth-supported=none --mon-host=127.0.0.1:7124 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/5 --osd-journal=td/osd-recovery-scrub/5/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.25813/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBvPv5dpBmECRAAvESm37WkFJ1GCk3TLzYrrg== --osd-uuid 7b76ca60-1092-4936-993f-4f50554de6dd 2019-12-21T15:46:55.501 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:46:55.499+0000 7f9edaa88ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T15:46:55.529 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:46:55.527+0000 7f9edaa88ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T15:46:55.531 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:46:55.528+0000 7f9edaa88ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T15:46:55.532 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:46:55.529+0000 7f9edaa88ec0 -1 bluestore(td/osd-recovery-scrub/5/block) _read_bdev_label failed to open td/osd-recovery-scrub/5/block: (2) No such file or directory 2019-12-21T15:46:55.532 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:46:55.529+0000 7f9edaa88ec0 -1 bluestore(td/osd-recovery-scrub/5/block) _read_bdev_label failed to open td/osd-recovery-scrub/5/block: (2) No such file or directory 2019-12-21T15:46:55.532 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:46:55.529+0000 7f9edaa88ec0 -1 bluestore(td/osd-recovery-scrub/5/block) _read_bdev_label failed to open td/osd-recovery-scrub/5/block: (2) No such file or directory 2019-12-21T15:46:55.587 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:46:55.585+0000 7f9edaa88ec0 -1 bluestore(td/osd-recovery-scrub/5) _read_fsid unparsable uuid 2019-12-21T15:46:58.600 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: local key_fn=td/osd-recovery-scrub/5/keyring 2019-12-21T15:46:58.600 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: cat 2019-12-21T15:46:58.601 INFO:tasks.workunit.client.0.smithi172.stdout:adding osd5 key to auth repository 2019-12-21T15:46:58.601 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: echo adding osd5 key to auth repository 2019-12-21T15:46:58.601 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph -i td/osd-recovery-scrub/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2019-12-21T15:46:58.891 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo start osd.5 2019-12-21T15:46:58.891 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph-osd -i 5 --fsid=72defa8a-f350-48c4-bd40-c303a855171e --auth-supported=none --mon-host=127.0.0.1:7124 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/5 --osd-journal=td/osd-recovery-scrub/5/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.25813/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2019-12-21T15:46:58.891 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph osd dump --format=json 2019-12-21T15:46:58.891 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: jq '.flags_set[]' 2019-12-21T15:46:58.892 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: grep -q '"noup"' 2019-12-21T15:46:58.892 INFO:tasks.workunit.client.0.smithi172.stdout:start osd.5 2019-12-21T15:46:58.909 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:46:58.906+0000 7f594e886ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T15:46:58.911 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:46:58.908+0000 7f594e886ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T15:46:58.913 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:46:58.909+0000 7f594e886ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T15:46:59.118 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: wait_for_osd up 5 2019-12-21T15:46:59.118 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:914: wait_for_osd: local state=up 2019-12-21T15:46:59.119 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local id=5 2019-12-21T15:46:59.119 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: status=1 2019-12-21T15:46:59.119 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i=0 )) 2019-12-21T15:46:59.119 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i < 300 )) 2019-12-21T15:46:59.119 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: echo 0 2019-12-21T15:46:59.119 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: ceph osd dump 2019-12-21T15:46:59.120 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: grep 'osd.5 up' 2019-12-21T15:46:59.120 INFO:tasks.workunit.client.0.smithi172.stdout:0 2019-12-21T15:46:59.342 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: sleep 1 2019-12-21T15:46:59.427 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:46:59.425+0000 7f594e886ec0 -1 Falling back to public interface 2019-12-21T15:47:00.131 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:47:00.128+0000 7f594e886ec0 -1 osd.5 0 log_to_monitors {default=true} 2019-12-21T15:47:00.344 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i++ )) 2019-12-21T15:47:00.344 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i < 300 )) 2019-12-21T15:47:00.344 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: echo 1 2019-12-21T15:47:00.345 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: ceph osd dump 2019-12-21T15:47:00.345 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: grep 'osd.5 up' 2019-12-21T15:47:00.345 INFO:tasks.workunit.client.0.smithi172.stdout:1 2019-12-21T15:47:00.637 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: sleep 1 2019-12-21T15:47:01.638 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i++ )) 2019-12-21T15:47:01.639 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i < 300 )) 2019-12-21T15:47:01.639 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: echo 2 2019-12-21T15:47:01.639 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: ceph osd dump 2019-12-21T15:47:01.639 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: grep 'osd.5 up' 2019-12-21T15:47:01.639 INFO:tasks.workunit.client.0.smithi172.stdout:2 2019-12-21T15:47:01.872 INFO:tasks.workunit.client.0.smithi172.stdout:osd.5 up in weight 1 up_from 29 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/28047,v1:127.0.0.1:6843/28047] [v2:127.0.0.1:6844/28047,v1:127.0.0.1:6845/28047] exists,up 7b76ca60-1092-4936-993f-4f50554de6dd 2019-12-21T15:47:01.873 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=0 2019-12-21T15:47:01.873 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: break 2019-12-21T15:47:01.873 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: return 0 2019-12-21T15:47:01.873 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:47: TEST_recovery_scrub: for osd in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:47:01.873 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:49: TEST_recovery_scrub: run_osd td/osd-recovery-scrub 6 2019-12-21T15:47:01.873 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local dir=td/osd-recovery-scrub 2019-12-21T15:47:01.874 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: shift 2019-12-21T15:47:01.874 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local id=6 2019-12-21T15:47:01.874 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2019-12-21T15:47:01.874 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local osd_data=td/osd-recovery-scrub/6 2019-12-21T15:47:01.874 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local 'ceph_args=--fsid=72defa8a-f350-48c4-bd40-c303a855171e --auth-supported=none --mon-host=127.0.0.1:7124 ' 2019-12-21T15:47:01.874 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2019-12-21T15:47:01.874 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-journal-size=100' 2019-12-21T15:47:01.875 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2019-12-21T15:47:01.875 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/6' 2019-12-21T15:47:01.875 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/6/journal' 2019-12-21T15:47:01.875 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --chdir=' 2019-12-21T15:47:01.875 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+= 2019-12-21T15:47:01.875 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2019-12-21T15:47:01.875 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: get_asok_path 2019-12-21T15:47:01.876 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2019-12-21T15:47:01.876 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2019-12-21T15:47:01.877 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2019-12-21T15:47:01.877 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:47:01.878 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.25813 2019-12-21T15:47:01.878 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.25813/$cluster-$name.asok' 2019-12-21T15:47:01.878 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.25813/$cluster-$name.asok' 2019-12-21T15:47:01.878 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --debug-osd=20' 2019-12-21T15:47:01.878 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-ms=1' 2019-12-21T15:47:01.878 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-monc=20' 2019-12-21T15:47:01.878 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2019-12-21T15:47:01.879 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2019-12-21T15:47:01.879 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2019-12-21T15:47:01.879 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2019-12-21T15:47:01.879 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2019-12-21T15:47:01.879 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' ' 2019-12-21T15:47:01.879 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+= 2019-12-21T15:47:01.879 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: mkdir -p td/osd-recovery-scrub/6 2019-12-21T15:47:01.880 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: uuidgen 2019-12-21T15:47:01.880 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: local uuid=ecafc269-5e15-403f-9def-ac6efc2c0917 2019-12-21T15:47:01.881 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: echo 'add osd6 ecafc269-5e15-403f-9def-ac6efc2c0917' 2019-12-21T15:47:01.881 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: ceph-authtool --gen-print-key 2019-12-21T15:47:01.881 INFO:tasks.workunit.client.0.smithi172.stdout:add osd6 ecafc269-5e15-403f-9def-ac6efc2c0917 2019-12-21T15:47:01.886 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: OSD_SECRET=AQB1Pv5deP+6NBAA8FIKt8ZbpMe3c8Egg1ENQg== 2019-12-21T15:47:01.886 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo '{"cephx_secret": "AQB1Pv5deP+6NBAA8FIKt8ZbpMe3c8Egg1ENQg=="}' 2019-12-21T15:47:01.886 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph osd new ecafc269-5e15-403f-9def-ac6efc2c0917 -i td/osd-recovery-scrub/6/new.json 2019-12-21T15:47:02.185 INFO:tasks.workunit.client.0.smithi172.stdout:6 2019-12-21T15:47:02.192 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: rm td/osd-recovery-scrub/6/new.json 2019-12-21T15:47:02.193 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-osd -i 6 --fsid=72defa8a-f350-48c4-bd40-c303a855171e --auth-supported=none --mon-host=127.0.0.1:7124 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/6 --osd-journal=td/osd-recovery-scrub/6/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.25813/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQB1Pv5deP+6NBAA8FIKt8ZbpMe3c8Egg1ENQg== --osd-uuid ecafc269-5e15-403f-9def-ac6efc2c0917 2019-12-21T15:47:02.212 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:47:02.209+0000 7f000a995ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T15:47:02.223 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:47:02.221+0000 7f000a995ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T15:47:02.225 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:47:02.223+0000 7f000a995ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T15:47:02.226 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:47:02.224+0000 7f000a995ec0 -1 bluestore(td/osd-recovery-scrub/6/block) _read_bdev_label failed to open td/osd-recovery-scrub/6/block: (2) No such file or directory 2019-12-21T15:47:02.227 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:47:02.224+0000 7f000a995ec0 -1 bluestore(td/osd-recovery-scrub/6/block) _read_bdev_label failed to open td/osd-recovery-scrub/6/block: (2) No such file or directory 2019-12-21T15:47:02.227 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:47:02.224+0000 7f000a995ec0 -1 bluestore(td/osd-recovery-scrub/6/block) _read_bdev_label failed to open td/osd-recovery-scrub/6/block: (2) No such file or directory 2019-12-21T15:47:02.282 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:47:02.280+0000 7f000a995ec0 -1 bluestore(td/osd-recovery-scrub/6) _read_fsid unparsable uuid 2019-12-21T15:47:05.334 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: local key_fn=td/osd-recovery-scrub/6/keyring 2019-12-21T15:47:05.334 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: cat 2019-12-21T15:47:05.335 INFO:tasks.workunit.client.0.smithi172.stdout:adding osd6 key to auth repository 2019-12-21T15:47:05.335 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: echo adding osd6 key to auth repository 2019-12-21T15:47:05.335 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph -i td/osd-recovery-scrub/6/keyring auth add osd.6 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2019-12-21T15:47:05.615 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo start osd.6 2019-12-21T15:47:05.616 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph osd dump --format=json 2019-12-21T15:47:05.616 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph-osd -i 6 --fsid=72defa8a-f350-48c4-bd40-c303a855171e --auth-supported=none --mon-host=127.0.0.1:7124 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/6 --osd-journal=td/osd-recovery-scrub/6/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.25813/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2019-12-21T15:47:05.616 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: jq '.flags_set[]' 2019-12-21T15:47:05.616 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: grep -q '"noup"' 2019-12-21T15:47:05.616 INFO:tasks.workunit.client.0.smithi172.stdout:start osd.6 2019-12-21T15:47:05.633 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:47:05.631+0000 7fa6b8f46ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T15:47:05.635 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:47:05.632+0000 7fa6b8f46ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T15:47:05.636 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:47:05.634+0000 7fa6b8f46ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T15:47:05.841 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: wait_for_osd up 6 2019-12-21T15:47:05.841 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:914: wait_for_osd: local state=up 2019-12-21T15:47:05.841 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local id=6 2019-12-21T15:47:05.841 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: status=1 2019-12-21T15:47:05.841 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i=0 )) 2019-12-21T15:47:05.842 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i < 300 )) 2019-12-21T15:47:05.842 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: echo 0 2019-12-21T15:47:05.842 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: ceph osd dump 2019-12-21T15:47:05.842 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: grep 'osd.6 up' 2019-12-21T15:47:05.842 INFO:tasks.workunit.client.0.smithi172.stdout:0 2019-12-21T15:47:06.066 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: sleep 1 2019-12-21T15:47:06.160 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:47:06.157+0000 7fa6b8f46ec0 -1 Falling back to public interface 2019-12-21T15:47:06.967 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:47:06.964+0000 7fa6b8f46ec0 -1 osd.6 0 log_to_monitors {default=true} 2019-12-21T15:47:07.068 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i++ )) 2019-12-21T15:47:07.068 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i < 300 )) 2019-12-21T15:47:07.068 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: echo 1 2019-12-21T15:47:07.069 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: ceph osd dump 2019-12-21T15:47:07.069 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: grep 'osd.6 up' 2019-12-21T15:47:07.069 INFO:tasks.workunit.client.0.smithi172.stdout:1 2019-12-21T15:47:07.338 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: sleep 1 2019-12-21T15:47:08.340 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i++ )) 2019-12-21T15:47:08.340 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i < 300 )) 2019-12-21T15:47:08.340 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: echo 2 2019-12-21T15:47:08.340 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: ceph osd dump 2019-12-21T15:47:08.340 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: grep 'osd.6 up' 2019-12-21T15:47:08.340 INFO:tasks.workunit.client.0.smithi172.stdout:2 2019-12-21T15:47:08.582 INFO:tasks.workunit.client.0.smithi172.stdout:osd.6 up in weight 1 up_from 34 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6850/28424,v1:127.0.0.1:6851/28424] [v2:127.0.0.1:6852/28424,v1:127.0.0.1:6853/28424] exists,up ecafc269-5e15-403f-9def-ac6efc2c0917 2019-12-21T15:47:08.583 INFO:tasks.workunit.client.0.smithi172.stdout:add osd7 d6e144d2-9a97-4c08-b545-b166fba9fdc0 2019-12-21T15:47:08.583 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=0 2019-12-21T15:47:08.583 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: break 2019-12-21T15:47:08.583 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: return 0 2019-12-21T15:47:08.583 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:47: TEST_recovery_scrub: for osd in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:47:08.584 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:49: TEST_recovery_scrub: run_osd td/osd-recovery-scrub 7 2019-12-21T15:47:08.584 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local dir=td/osd-recovery-scrub 2019-12-21T15:47:08.584 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: shift 2019-12-21T15:47:08.584 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local id=7 2019-12-21T15:47:08.584 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2019-12-21T15:47:08.584 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local osd_data=td/osd-recovery-scrub/7 2019-12-21T15:47:08.584 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local 'ceph_args=--fsid=72defa8a-f350-48c4-bd40-c303a855171e --auth-supported=none --mon-host=127.0.0.1:7124 ' 2019-12-21T15:47:08.585 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2019-12-21T15:47:08.585 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-journal-size=100' 2019-12-21T15:47:08.585 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2019-12-21T15:47:08.585 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/7' 2019-12-21T15:47:08.585 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/7/journal' 2019-12-21T15:47:08.585 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --chdir=' 2019-12-21T15:47:08.586 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+= 2019-12-21T15:47:08.586 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2019-12-21T15:47:08.586 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: get_asok_path 2019-12-21T15:47:08.586 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2019-12-21T15:47:08.586 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2019-12-21T15:47:08.586 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2019-12-21T15:47:08.586 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:47:08.586 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.25813 2019-12-21T15:47:08.587 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.25813/$cluster-$name.asok' 2019-12-21T15:47:08.587 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.25813/$cluster-$name.asok' 2019-12-21T15:47:08.587 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --debug-osd=20' 2019-12-21T15:47:08.587 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-ms=1' 2019-12-21T15:47:08.587 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-monc=20' 2019-12-21T15:47:08.587 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2019-12-21T15:47:08.587 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2019-12-21T15:47:08.588 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2019-12-21T15:47:08.588 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2019-12-21T15:47:08.588 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2019-12-21T15:47:08.588 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' ' 2019-12-21T15:47:08.589 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+= 2019-12-21T15:47:08.589 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: mkdir -p td/osd-recovery-scrub/7 2019-12-21T15:47:08.589 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: uuidgen 2019-12-21T15:47:08.590 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: local uuid=d6e144d2-9a97-4c08-b545-b166fba9fdc0 2019-12-21T15:47:08.590 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: echo 'add osd7 d6e144d2-9a97-4c08-b545-b166fba9fdc0' 2019-12-21T15:47:08.590 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: ceph-authtool --gen-print-key 2019-12-21T15:47:08.591 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: OSD_SECRET=AQB8Pv5d/ST7IhAA65gegpl39/ihYe3xqZpA7A== 2019-12-21T15:47:08.591 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo '{"cephx_secret": "AQB8Pv5d/ST7IhAA65gegpl39/ihYe3xqZpA7A=="}' 2019-12-21T15:47:08.591 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph osd new d6e144d2-9a97-4c08-b545-b166fba9fdc0 -i td/osd-recovery-scrub/7/new.json 2019-12-21T15:47:08.917 INFO:tasks.workunit.client.0.smithi172.stdout:7 2019-12-21T15:47:08.926 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: rm td/osd-recovery-scrub/7/new.json 2019-12-21T15:47:08.927 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-osd -i 7 --fsid=72defa8a-f350-48c4-bd40-c303a855171e --auth-supported=none --mon-host=127.0.0.1:7124 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/7 --osd-journal=td/osd-recovery-scrub/7/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.25813/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQB8Pv5d/ST7IhAA65gegpl39/ihYe3xqZpA7A== --osd-uuid d6e144d2-9a97-4c08-b545-b166fba9fdc0 2019-12-21T15:47:08.948 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:47:08.945+0000 7f3e96b9aec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T15:47:08.949 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:47:08.947+0000 7f3e96b9aec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T15:47:08.951 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:47:08.948+0000 7f3e96b9aec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T15:47:08.952 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:47:08.949+0000 7f3e96b9aec0 -1 bluestore(td/osd-recovery-scrub/7/block) _read_bdev_label failed to open td/osd-recovery-scrub/7/block: (2) No such file or directory 2019-12-21T15:47:08.952 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:47:08.949+0000 7f3e96b9aec0 -1 bluestore(td/osd-recovery-scrub/7/block) _read_bdev_label failed to open td/osd-recovery-scrub/7/block: (2) No such file or directory 2019-12-21T15:47:08.952 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:47:08.949+0000 7f3e96b9aec0 -1 bluestore(td/osd-recovery-scrub/7/block) _read_bdev_label failed to open td/osd-recovery-scrub/7/block: (2) No such file or directory 2019-12-21T15:47:09.041 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:47:09.039+0000 7f3e96b9aec0 -1 bluestore(td/osd-recovery-scrub/7) _read_fsid unparsable uuid 2019-12-21T15:47:12.127 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: local key_fn=td/osd-recovery-scrub/7/keyring 2019-12-21T15:47:12.128 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: cat 2019-12-21T15:47:12.129 INFO:tasks.workunit.client.0.smithi172.stdout:adding osd7 key to auth repository 2019-12-21T15:47:12.129 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: echo adding osd7 key to auth repository 2019-12-21T15:47:12.129 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph -i td/osd-recovery-scrub/7/keyring auth add osd.7 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2019-12-21T15:47:12.414 INFO:tasks.workunit.client.0.smithi172.stdout:start osd.7 2019-12-21T15:47:12.414 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo start osd.7 2019-12-21T15:47:12.414 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph-osd -i 7 --fsid=72defa8a-f350-48c4-bd40-c303a855171e --auth-supported=none --mon-host=127.0.0.1:7124 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/7 --osd-journal=td/osd-recovery-scrub/7/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.25813/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2019-12-21T15:47:12.414 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph osd dump --format=json 2019-12-21T15:47:12.414 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: jq '.flags_set[]' 2019-12-21T15:47:12.415 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: grep -q '"noup"' 2019-12-21T15:47:12.431 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:47:12.430+0000 7fb8c7904ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T15:47:12.433 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:47:12.432+0000 7fb8c7904ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T15:47:12.435 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:47:12.433+0000 7fb8c7904ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T15:47:12.643 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: wait_for_osd up 7 2019-12-21T15:47:12.643 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:914: wait_for_osd: local state=up 2019-12-21T15:47:12.643 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local id=7 2019-12-21T15:47:12.643 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: status=1 2019-12-21T15:47:12.643 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i=0 )) 2019-12-21T15:47:12.644 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i < 300 )) 2019-12-21T15:47:12.644 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: echo 0 2019-12-21T15:47:12.644 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: ceph osd dump 2019-12-21T15:47:12.644 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: grep 'osd.7 up' 2019-12-21T15:47:12.644 INFO:tasks.workunit.client.0.smithi172.stdout:0 2019-12-21T15:47:12.868 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: sleep 1 2019-12-21T15:47:12.951 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:47:12.949+0000 7fb8c7904ec0 -1 Falling back to public interface 2019-12-21T15:47:13.718 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:47:13.717+0000 7fb8c7904ec0 -1 osd.7 0 log_to_monitors {default=true} 2019-12-21T15:47:13.870 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i++ )) 2019-12-21T15:47:13.870 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i < 300 )) 2019-12-21T15:47:13.870 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: echo 1 2019-12-21T15:47:13.870 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: ceph osd dump 2019-12-21T15:47:13.870 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: grep 'osd.7 up' 2019-12-21T15:47:13.871 INFO:tasks.workunit.client.0.smithi172.stdout:1 2019-12-21T15:47:14.114 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: sleep 1 2019-12-21T15:47:15.116 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i++ )) 2019-12-21T15:47:15.116 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i < 300 )) 2019-12-21T15:47:15.116 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: echo 2 2019-12-21T15:47:15.117 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: ceph osd dump 2019-12-21T15:47:15.117 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: grep 'osd.7 up' 2019-12-21T15:47:15.117 INFO:tasks.workunit.client.0.smithi172.stdout:2 2019-12-21T15:47:15.351 INFO:tasks.workunit.client.0.smithi172.stdout:osd.7 up in weight 1 up_from 39 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6858/28801,v1:127.0.0.1:6859/28801] [v2:127.0.0.1:6860/28801,v1:127.0.0.1:6861/28801] exists,up d6e144d2-9a97-4c08-b545-b166fba9fdc0 2019-12-21T15:47:15.351 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=0 2019-12-21T15:47:15.352 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: break 2019-12-21T15:47:15.352 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: return 0 2019-12-21T15:47:15.352 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:53: TEST_recovery_scrub: create_pool test 32 32 2019-12-21T15:47:15.352 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:539: create_pool: ceph osd pool create test 32 32 2019-12-21T15:47:15.682 INFO:tasks.workunit.client.0.smithi172.stderr:pool 'test' created 2019-12-21T15:47:15.689 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: sleep 1 2019-12-21T15:47:16.691 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:54: TEST_recovery_scrub: wait_for_clean 2019-12-21T15:47:16.691 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1548: wait_for_clean: local cmd= 2019-12-21T15:47:16.691 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1549: wait_for_clean: local num_active_clean=-1 2019-12-21T15:47:16.691 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1550: wait_for_clean: local cur_active_clean 2019-12-21T15:47:16.692 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1551: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2019-12-21T15:47:16.693 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1551: wait_for_clean: get_timeout_delays 90 .1 2019-12-21T15:47:16.693 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1494: get_timeout_delays: shopt -q -o xtrace 2019-12-21T15:47:16.693 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1494: get_timeout_delays: echo true 2019-12-21T15:47:16.693 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1494: get_timeout_delays: local trace=true 2019-12-21T15:47:16.693 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1495: get_timeout_delays: true 2019-12-21T15:47:16.693 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1495: get_timeout_delays: shopt -u -o xtrace 2019-12-21T15:47:16.783 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1551: wait_for_clean: local -a delays 2019-12-21T15:47:16.783 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1552: wait_for_clean: local -i loop=0 2019-12-21T15:47:16.783 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1554: wait_for_clean: flush_pg_stats 2019-12-21T15:47:16.784 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1977: flush_pg_stats: local timeout=300 2019-12-21T15:47:16.784 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1979: flush_pg_stats: ceph osd ls 2019-12-21T15:47:17.079 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1979: flush_pg_stats: ids='0 2019-12-21T15:47:17.079 INFO:tasks.workunit.client.0.smithi172.stderr:1 2019-12-21T15:47:17.079 INFO:tasks.workunit.client.0.smithi172.stderr:2 2019-12-21T15:47:17.079 INFO:tasks.workunit.client.0.smithi172.stderr:3 2019-12-21T15:47:17.079 INFO:tasks.workunit.client.0.smithi172.stderr:4 2019-12-21T15:47:17.079 INFO:tasks.workunit.client.0.smithi172.stderr:5 2019-12-21T15:47:17.080 INFO:tasks.workunit.client.0.smithi172.stderr:6 2019-12-21T15:47:17.080 INFO:tasks.workunit.client.0.smithi172.stderr:7' 2019-12-21T15:47:17.080 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1980: flush_pg_stats: seqs= 2019-12-21T15:47:17.080 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: flush_pg_stats: for osd in $ids 2019-12-21T15:47:17.080 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2019-12-21T15:47:17.189 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: seq=21474836492 2019-12-21T15:47:17.189 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: flush_pg_stats: seqs=' 0-21474836492' 2019-12-21T15:47:17.190 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: flush_pg_stats: for osd in $ids 2019-12-21T15:47:17.190 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2019-12-21T15:47:17.303 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: seq=42949672970 2019-12-21T15:47:17.304 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: flush_pg_stats: seqs=' 0-21474836492 1-42949672970' 2019-12-21T15:47:17.304 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: flush_pg_stats: for osd in $ids 2019-12-21T15:47:17.304 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2019-12-21T15:47:17.410 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: seq=64424509449 2019-12-21T15:47:17.410 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: flush_pg_stats: seqs=' 0-21474836492 1-42949672970 2-64424509449' 2019-12-21T15:47:17.411 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: flush_pg_stats: for osd in $ids 2019-12-21T15:47:17.411 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2019-12-21T15:47:17.520 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: seq=85899345928 2019-12-21T15:47:17.520 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: flush_pg_stats: seqs=' 0-21474836492 1-42949672970 2-64424509449 3-85899345928' 2019-12-21T15:47:17.520 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: flush_pg_stats: for osd in $ids 2019-12-21T15:47:17.523 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2019-12-21T15:47:17.628 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: seq=107374182406 2019-12-21T15:47:17.628 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: flush_pg_stats: seqs=' 0-21474836492 1-42949672970 2-64424509449 3-85899345928 4-107374182406' 2019-12-21T15:47:17.628 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: flush_pg_stats: for osd in $ids 2019-12-21T15:47:17.629 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2019-12-21T15:47:17.739 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: seq=124554051589 2019-12-21T15:47:17.739 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: flush_pg_stats: seqs=' 0-21474836492 1-42949672970 2-64424509449 3-85899345928 4-107374182406 5-124554051589' 2019-12-21T15:47:17.739 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: flush_pg_stats: for osd in $ids 2019-12-21T15:47:17.740 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2019-12-21T15:47:17.850 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: seq=146028888067 2019-12-21T15:47:17.850 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: flush_pg_stats: seqs=' 0-21474836492 1-42949672970 2-64424509449 3-85899345928 4-107374182406 5-124554051589 6-146028888067' 2019-12-21T15:47:17.850 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: flush_pg_stats: for osd in $ids 2019-12-21T15:47:17.851 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2019-12-21T15:47:17.958 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: seq=167503724546 2019-12-21T15:47:17.958 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: flush_pg_stats: seqs=' 0-21474836492 1-42949672970 2-64424509449 3-85899345928 4-107374182406 5-124554051589 6-146028888067 7-167503724546' 2019-12-21T15:47:17.958 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: flush_pg_stats: for s in $seqs 2019-12-21T15:47:17.959 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: echo 0-21474836492 2019-12-21T15:47:17.959 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: cut -d - -f 1 2019-12-21T15:47:17.959 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: osd=0 2019-12-21T15:47:17.961 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: echo 0-21474836492 2019-12-21T15:47:17.961 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: cut -d - -f 2 2019-12-21T15:47:17.962 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: seq=21474836492 2019-12-21T15:47:17.962 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1989: flush_pg_stats: echo 'waiting osd.0 seq 21474836492' 2019-12-21T15:47:17.962 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.0 seq 21474836492 2019-12-21T15:47:17.963 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 0 2019-12-21T15:47:18.188 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 21474836491 -lt 21474836492 2019-12-21T15:47:18.188 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1991: flush_pg_stats: sleep 1 2019-12-21T15:47:19.189 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1992: flush_pg_stats: '[' 300 -eq 0 ']' 2019-12-21T15:47:19.190 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 0 2019-12-21T15:47:19.427 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 21474836493 -lt 21474836492 2019-12-21T15:47:19.427 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: flush_pg_stats: for s in $seqs 2019-12-21T15:47:19.428 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: echo 1-42949672970 2019-12-21T15:47:19.428 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: cut -d - -f 1 2019-12-21T15:47:19.428 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: osd=1 2019-12-21T15:47:19.429 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: echo 1-42949672970 2019-12-21T15:47:19.430 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: cut -d - -f 2 2019-12-21T15:47:19.431 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: seq=42949672970 2019-12-21T15:47:19.431 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1989: flush_pg_stats: echo 'waiting osd.1 seq 42949672970' 2019-12-21T15:47:19.431 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.1 seq 42949672970 2019-12-21T15:47:19.431 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 1 2019-12-21T15:47:19.656 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 42949672970 -lt 42949672970 2019-12-21T15:47:19.657 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: flush_pg_stats: for s in $seqs 2019-12-21T15:47:19.657 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: echo 2-64424509449 2019-12-21T15:47:19.657 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: cut -d - -f 1 2019-12-21T15:47:19.658 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: osd=2 2019-12-21T15:47:19.659 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: echo 2-64424509449 2019-12-21T15:47:19.659 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: cut -d - -f 2 2019-12-21T15:47:19.660 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: seq=64424509449 2019-12-21T15:47:19.660 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1989: flush_pg_stats: echo 'waiting osd.2 seq 64424509449' 2019-12-21T15:47:19.660 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.2 seq 64424509449 2019-12-21T15:47:19.661 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 2 2019-12-21T15:47:19.887 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 64424509449 -lt 64424509449 2019-12-21T15:47:19.887 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: flush_pg_stats: for s in $seqs 2019-12-21T15:47:19.888 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: echo 3-85899345928 2019-12-21T15:47:19.888 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: cut -d - -f 1 2019-12-21T15:47:19.889 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: osd=3 2019-12-21T15:47:19.890 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: echo 3-85899345928 2019-12-21T15:47:19.890 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: cut -d - -f 2 2019-12-21T15:47:19.891 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: seq=85899345928 2019-12-21T15:47:19.891 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1989: flush_pg_stats: echo 'waiting osd.3 seq 85899345928' 2019-12-21T15:47:19.891 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.3 seq 85899345928 2019-12-21T15:47:19.891 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 3 2019-12-21T15:47:20.117 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 85899345929 -lt 85899345928 2019-12-21T15:47:20.117 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: flush_pg_stats: for s in $seqs 2019-12-21T15:47:20.118 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: echo 4-107374182406 2019-12-21T15:47:20.118 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: cut -d - -f 1 2019-12-21T15:47:20.119 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: osd=4 2019-12-21T15:47:20.120 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: echo 4-107374182406 2019-12-21T15:47:20.120 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: cut -d - -f 2 2019-12-21T15:47:20.124 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: seq=107374182406 2019-12-21T15:47:20.124 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1989: flush_pg_stats: echo 'waiting osd.4 seq 107374182406' 2019-12-21T15:47:20.124 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 4 2019-12-21T15:47:20.124 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.4 seq 107374182406 2019-12-21T15:47:20.345 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 107374182406 -lt 107374182406 2019-12-21T15:47:20.346 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: flush_pg_stats: for s in $seqs 2019-12-21T15:47:20.346 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: echo 5-124554051589 2019-12-21T15:47:20.346 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: cut -d - -f 1 2019-12-21T15:47:20.347 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: osd=5 2019-12-21T15:47:20.348 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: echo 5-124554051589 2019-12-21T15:47:20.348 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: cut -d - -f 2 2019-12-21T15:47:20.349 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: seq=124554051589 2019-12-21T15:47:20.349 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1989: flush_pg_stats: echo 'waiting osd.5 seq 124554051589' 2019-12-21T15:47:20.349 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.5 seq 124554051589 2019-12-21T15:47:20.350 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 5 2019-12-21T15:47:20.574 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 124554051589 -lt 124554051589 2019-12-21T15:47:20.574 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: flush_pg_stats: for s in $seqs 2019-12-21T15:47:20.575 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: echo 6-146028888067 2019-12-21T15:47:20.575 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: cut -d - -f 1 2019-12-21T15:47:20.576 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: osd=6 2019-12-21T15:47:20.577 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: echo 6-146028888067 2019-12-21T15:47:20.577 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: cut -d - -f 2 2019-12-21T15:47:20.577 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: seq=146028888067 2019-12-21T15:47:20.578 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1989: flush_pg_stats: echo 'waiting osd.6 seq 146028888067' 2019-12-21T15:47:20.578 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.6 seq 146028888067 2019-12-21T15:47:20.578 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 6 2019-12-21T15:47:20.802 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 146028888068 -lt 146028888067 2019-12-21T15:47:20.803 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: flush_pg_stats: for s in $seqs 2019-12-21T15:47:20.803 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: echo 7-167503724546 2019-12-21T15:47:20.803 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: cut -d - -f 1 2019-12-21T15:47:20.804 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: osd=7 2019-12-21T15:47:20.805 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: echo 7-167503724546 2019-12-21T15:47:20.805 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: cut -d - -f 2 2019-12-21T15:47:20.806 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: seq=167503724546 2019-12-21T15:47:20.806 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1989: flush_pg_stats: echo 'waiting osd.7 seq 167503724546' 2019-12-21T15:47:20.806 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.7 seq 167503724546 2019-12-21T15:47:20.807 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 7 2019-12-21T15:47:21.033 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 167503724547 -lt 167503724546 2019-12-21T15:47:21.033 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1555: wait_for_clean: get_num_pgs 2019-12-21T15:47:21.034 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: ceph --format json status 2019-12-21T15:47:21.034 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: jq .pgmap.num_pgs 2019-12-21T15:47:21.302 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1555: wait_for_clean: test 32 == 0 2019-12-21T15:47:21.302 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1559: wait_for_clean: true 2019-12-21T15:47:21.303 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1563: wait_for_clean: get_num_active_clean 2019-12-21T15:47:21.303 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: get_num_active_clean: local expression 2019-12-21T15:47:21.303 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2019-12-21T15:47:21.303 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("stale") | not)' 2019-12-21T15:47:21.303 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: ceph --format json pg dump pgs 2019-12-21T15:47:21.303 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2019-12-21T15:47:21.527 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1563: wait_for_clean: cur_active_clean=32 2019-12-21T15:47:21.528 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1564: wait_for_clean: get_num_pgs 2019-12-21T15:47:21.529 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: ceph --format json status 2019-12-21T15:47:21.529 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: jq .pgmap.num_pgs 2019-12-21T15:47:21.797 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1564: wait_for_clean: test 32 = 32 2019-12-21T15:47:21.797 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1564: wait_for_clean: break 2019-12-21T15:47:21.797 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: return 0 2019-12-21T15:47:21.798 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:55: TEST_recovery_scrub: ceph osd dump 2019-12-21T15:47:21.798 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:55: TEST_recovery_scrub: grep '^pool.*['\'']test['\'']' 2019-12-21T15:47:21.798 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:55: TEST_recovery_scrub: awk '{ print $2 }' 2019-12-21T15:47:22.025 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:55: TEST_recovery_scrub: poolid=1 2019-12-21T15:47:22.025 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:57: TEST_recovery_scrub: dd if=/dev/urandom of=testdata.25813 bs=1M count=50 2019-12-21T15:47:22.253 INFO:tasks.workunit.client.0.smithi172.stderr:50+0 records in 2019-12-21T15:47:22.253 INFO:tasks.workunit.client.0.smithi172.stderr:50+0 records out 2019-12-21T15:47:22.254 INFO:tasks.workunit.client.0.smithi172.stderr:52428800 bytes (52 MB, 50 MiB) copied, 0.227615 s, 230 MB/s 2019-12-21T15:47:22.254 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:58: TEST_recovery_scrub: seq 1 4 2019-12-21T15:47:22.255 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:58: TEST_recovery_scrub: for i in $(seq 1 $OBJECTS) 2019-12-21T15:47:22.255 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:60: TEST_recovery_scrub: rados -p test put obj1 testdata.25813 2019-12-21T15:47:23.609 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:58: TEST_recovery_scrub: for i in $(seq 1 $OBJECTS) 2019-12-21T15:47:23.609 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:60: TEST_recovery_scrub: rados -p test put obj2 testdata.25813 2019-12-21T15:47:24.955 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:58: TEST_recovery_scrub: for i in $(seq 1 $OBJECTS) 2019-12-21T15:47:24.955 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:60: TEST_recovery_scrub: rados -p test put obj3 testdata.25813 2019-12-21T15:47:26.168 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:58: TEST_recovery_scrub: for i in $(seq 1 $OBJECTS) 2019-12-21T15:47:26.169 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:60: TEST_recovery_scrub: rados -p test put obj4 testdata.25813 2019-12-21T15:47:27.456 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:62: TEST_recovery_scrub: rm -f testdata.25813 2019-12-21T15:47:27.463 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:64: TEST_recovery_scrub: ceph osd pool set test size 4 2019-12-21T15:47:27.801 INFO:tasks.workunit.client.0.smithi172.stderr:set pool 1 size to 4 2019-12-21T15:47:27.808 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:66: TEST_recovery_scrub: pids= 2019-12-21T15:47:27.809 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:67: TEST_recovery_scrub: expr 32 - 1 2019-12-21T15:47:27.810 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:67: TEST_recovery_scrub: seq 0 31 2019-12-21T15:47:27.811 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:67: TEST_recovery_scrub: for pg in $(seq 0 $(expr $PGS - 1)) 2019-12-21T15:47:27.813 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:69: TEST_recovery_scrub: printf %x 0 2019-12-21T15:47:27.813 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:69: TEST_recovery_scrub: run_in_background pids pg_scrub 1.0 2019-12-21T15:47:27.813 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1907: run_in_background: local pid_variable=pids 2019-12-21T15:47:27.813 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1908: run_in_background: shift 2019-12-21T15:47:27.814 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1912: run_in_background: eval 'pids+=" 29863"' 2019-12-21T15:47:27.814 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1912: run_in_background: pids+=' 29863' 2019-12-21T15:47:27.814 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:67: TEST_recovery_scrub: for pg in $(seq 0 $(expr $PGS - 1)) 2019-12-21T15:47:27.814 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1911: run_in_background: pg_scrub 1.0 2019-12-21T15:47:27.814 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1911: run_in_background: sed 's/^/25813: /' 2019-12-21T15:47:27.814 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:69: TEST_recovery_scrub: printf %x 1 2019-12-21T15:47:27.817 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:69: TEST_recovery_scrub: run_in_background pids pg_scrub 1.1 2019-12-21T15:47:27.817 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1907: run_in_background: local pid_variable=pids 2019-12-21T15:47:27.817 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1908: run_in_background: shift 2019-12-21T15:47:27.817 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1912: run_in_background: eval 'pids+=" 29870"' 2019-12-21T15:47:27.818 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1912: run_in_background: pids+=' 29870' 2019-12-21T15:47:27.818 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:67: TEST_recovery_scrub: for pg in $(seq 0 $(expr $PGS - 1)) 2019-12-21T15:47:27.818 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1911: run_in_background: pg_scrub 1.1 2019-12-21T15:47:27.818 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1911: run_in_background: sed 's/^/25813: /' 2019-12-21T15:47:27.818 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:69: TEST_recovery_scrub: printf %x 2 2019-12-21T15:47:27.818 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:69: TEST_recovery_scrub: run_in_background pids pg_scrub 1.2 2019-12-21T15:47:27.818 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1907: run_in_background: local pid_variable=pids 2019-12-21T15:47:27.819 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1908: run_in_background: shift 2019-12-21T15:47:27.819 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1912: run_in_background: eval 'pids+=" 29877"' 2019-12-21T15:47:27.819 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1912: run_in_background: pids+=' 29877' 2019-12-21T15:47:27.819 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:67: TEST_recovery_scrub: for pg in $(seq 0 $(expr $PGS - 1)) 2019-12-21T15:47:27.819 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1911: run_in_background: pg_scrub 1.2 2019-12-21T15:47:27.819 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1911: run_in_background: sed 's/^/25813: /' 2019-12-21T15:47:27.819 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:69: TEST_recovery_scrub: printf %x 3 2019-12-21T15:47:27.820 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:69: TEST_recovery_scrub: run_in_background pids pg_scrub 1.3 2019-12-21T15:47:27.820 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1907: run_in_background: local pid_variable=pids 2019-12-21T15:47:27.820 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1908: run_in_background: shift 2019-12-21T15:47:27.820 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1912: run_in_background: eval 'pids+=" 29884"' 2019-12-21T15:47:27.820 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1912: run_in_background: pids+=' 29884' 2019-12-21T15:47:27.820 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:67: TEST_recovery_scrub: for pg in $(seq 0 $(expr $PGS - 1)) 2019-12-21T15:47:27.823 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1911: run_in_background: pg_scrub 1.3 2019-12-21T15:47:27.823 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1911: run_in_background: sed 's/^/25813: /' 2019-12-21T15:47:27.823 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:69: TEST_recovery_scrub: printf %x 4 2019-12-21T15:47:27.823 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:69: TEST_recovery_scrub: run_in_background pids pg_scrub 1.4 2019-12-21T15:47:27.823 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1907: run_in_background: local pid_variable=pids 2019-12-21T15:47:27.824 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1908: run_in_background: shift 2019-12-21T15:47:27.824 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1912: run_in_background: eval 'pids+=" 29891"' 2019-12-21T15:47:27.824 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1912: run_in_background: pids+=' 29891' 2019-12-21T15:47:27.824 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:67: TEST_recovery_scrub: for pg in $(seq 0 $(expr $PGS - 1)) 2019-12-21T15:47:27.824 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1911: run_in_background: pg_scrub 1.4 2019-12-21T15:47:27.824 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:69: TEST_recovery_scrub: printf %x 5 2019-12-21T15:47:27.824 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:69: TEST_recovery_scrub: run_in_background pids pg_scrub 1.5 2019-12-21T15:47:27.825 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1907: run_in_background: local pid_variable=pids 2019-12-21T15:47:27.825 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1908: run_in_background: shift 2019-12-21T15:47:27.825 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1912: run_in_background: eval 'pids+=" 29898"' 2019-12-21T15:47:27.825 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1912: run_in_background: pids+=' 29898' 2019-12-21T15:47:27.825 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:67: TEST_recovery_scrub: for pg in $(seq 0 $(expr $PGS - 1)) 2019-12-21T15:47:27.825 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1911: run_in_background: sed 's/^/25813: /' 2019-12-21T15:47:27.825 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:69: TEST_recovery_scrub: printf %x 6 2019-12-21T15:47:27.826 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:69: TEST_recovery_scrub: run_in_background pids pg_scrub 1.6 2019-12-21T15:47:27.826 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1907: run_in_background: local pid_variable=pids 2019-12-21T15:47:27.826 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1908: run_in_background: shift 2019-12-21T15:47:27.826 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1912: run_in_background: eval 'pids+=" 29902"' 2019-12-21T15:47:27.826 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1912: run_in_background: pids+=' 29902' 2019-12-21T15:47:27.826 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:67: TEST_recovery_scrub: for pg in $(seq 0 $(expr $PGS - 1)) 2019-12-21T15:47:27.827 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1911: run_in_background: pg_scrub 1.5 2019-12-21T15:47:27.827 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1911: run_in_background: sed 's/^/25813: /' 2019-12-21T15:47:27.827 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1911: run_in_background: pg_scrub 1.6 2019-12-21T15:47:27.828 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:69: TEST_recovery_scrub: printf %x 7 2019-12-21T15:47:27.828 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:69: TEST_recovery_scrub: run_in_background pids pg_scrub 1.7 2019-12-21T15:47:27.828 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1907: run_in_background: local pid_variable=pids 2019-12-21T15:47:27.828 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1908: run_in_background: shift 2019-12-21T15:47:27.828 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1912: run_in_background: eval 'pids+=" 29910"' 2019-12-21T15:47:27.828 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1912: run_in_background: pids+=' 29910' 2019-12-21T15:47:27.829 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:67: TEST_recovery_scrub: for pg in $(seq 0 $(expr $PGS - 1)) 2019-12-21T15:47:27.829 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:69: TEST_recovery_scrub: printf %x 8 2019-12-21T15:47:27.830 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:69: TEST_recovery_scrub: run_in_background pids pg_scrub 1.8 2019-12-21T15:47:27.830 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1907: run_in_background: local pid_variable=pids 2019-12-21T15:47:27.830 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1908: run_in_background: shift 2019-12-21T15:47:27.830 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1912: run_in_background: eval 'pids+=" 29912"' 2019-12-21T15:47:27.830 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1912: run_in_background: pids+=' 29912' 2019-12-21T15:47:27.830 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:67: TEST_recovery_scrub: for pg in $(seq 0 $(expr $PGS - 1)) 2019-12-21T15:47:27.830 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1911: run_in_background: sed 's/^/25813: /' 2019-12-21T15:47:27.831 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:69: TEST_recovery_scrub: printf %x 9 2019-12-21T15:47:27.831 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1911: run_in_background: sed 's/^/25813: /' 2019-12-21T15:47:27.833 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:69: TEST_recovery_scrub: run_in_background pids pg_scrub 1.9 2019-12-21T15:47:27.833 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1907: run_in_background: local pid_variable=pids 2019-12-21T15:47:27.833 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1908: run_in_background: shift 2019-12-21T15:47:27.833 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1912: run_in_background: eval 'pids+=" 29918"' 2019-12-21T15:47:27.833 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1912: run_in_background: pids+=' 29918' 2019-12-21T15:47:27.833 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:67: TEST_recovery_scrub: for pg in $(seq 0 $(expr $PGS - 1)) 2019-12-21T15:47:27.834 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1911: run_in_background: sed 's/^/25813: /' 2019-12-21T15:47:27.834 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:69: TEST_recovery_scrub: printf %x 10 2019-12-21T15:47:27.835 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:69: TEST_recovery_scrub: run_in_background pids pg_scrub 1.a 2019-12-21T15:47:27.835 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1907: run_in_background: local pid_variable=pids 2019-12-21T15:47:27.835 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1908: run_in_background: shift 2019-12-21T15:47:27.835 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1912: run_in_background: eval 'pids+=" 29922"' 2019-12-21T15:47:27.835 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1912: run_in_background: pids+=' 29922' 2019-12-21T15:47:27.836 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:67: TEST_recovery_scrub: for pg in $(seq 0 $(expr $PGS - 1)) 2019-12-21T15:47:27.836 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:69: TEST_recovery_scrub: printf %x 11 2019-12-21T15:47:27.837 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:69: TEST_recovery_scrub: run_in_background pids pg_scrub 1.b 2019-12-21T15:47:27.837 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1907: run_in_background: local pid_variable=pids 2019-12-21T15:47:27.837 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1908: run_in_background: shift 2019-12-21T15:47:27.837 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1912: run_in_background: eval 'pids+=" 29924"' 2019-12-21T15:47:27.838 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1912: run_in_background: pids+=' 29924' 2019-12-21T15:47:27.838 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:67: TEST_recovery_scrub: for pg in $(seq 0 $(expr $PGS - 1)) 2019-12-21T15:47:27.838 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1911: run_in_background: pg_scrub 1.b 2019-12-21T15:47:27.843 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1911: run_in_background: pg_scrub 1.8 2019-12-21T15:47:27.852 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1911: run_in_background: pg_scrub 1.9 2019-12-21T15:47:27.852 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1911: run_in_background: sed 's/^/25813: /' 2019-12-21T15:47:27.852 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1911: run_in_background: sed 's/^/25813: /' 2019-12-21T15:47:27.854 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1911: run_in_background: sed 's/^/25813: /' 2019-12-21T15:47:27.856 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1911: run_in_background: pg_scrub 1.a 2019-12-21T15:47:27.856 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1911: run_in_background: pg_scrub 1.7 2019-12-21T15:47:27.856 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:69: TEST_recovery_scrub: printf %x 12 2019-12-21T15:47:27.857 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:69: TEST_recovery_scrub: run_in_background pids pg_scrub 1.c 2019-12-21T15:47:27.857 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1907: run_in_background: local pid_variable=pids 2019-12-21T15:47:27.857 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1908: run_in_background: shift 2019-12-21T15:47:27.858 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1912: run_in_background: eval 'pids+=" 29941"' 2019-12-21T15:47:27.858 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1912: run_in_background: pids+=' 29941' 2019-12-21T15:47:27.858 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:67: TEST_recovery_scrub: for pg in $(seq 0 $(expr $PGS - 1)) 2019-12-21T15:47:27.867 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:69: TEST_recovery_scrub: printf %x 13 2019-12-21T15:47:27.868 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:69: TEST_recovery_scrub: run_in_background pids pg_scrub 1.d 2019-12-21T15:47:27.868 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1907: run_in_background: local pid_variable=pids 2019-12-21T15:47:27.869 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1908: run_in_background: shift 2019-12-21T15:47:27.869 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1912: run_in_background: eval 'pids+=" 29949"' 2019-12-21T15:47:27.869 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1912: run_in_background: pids+=' 29949' 2019-12-21T15:47:27.869 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:67: TEST_recovery_scrub: for pg in $(seq 0 $(expr $PGS - 1)) 2019-12-21T15:47:27.879 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1911: run_in_background: pg_scrub 1.c 2019-12-21T15:47:27.881 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1911: run_in_background: sed 's/^/25813: /' 2019-12-21T15:47:27.883 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:69: TEST_recovery_scrub: printf %x 14 2019-12-21T15:47:27.884 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:69: TEST_recovery_scrub: run_in_background pids pg_scrub 1.e 2019-12-21T15:47:27.884 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1907: run_in_background: local pid_variable=pids 2019-12-21T15:47:27.884 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1908: run_in_background: shift 2019-12-21T15:47:27.884 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1912: run_in_background: eval 'pids+=" 29956"' 2019-12-21T15:47:27.885 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1912: run_in_background: pids+=' 29956' 2019-12-21T15:47:27.885 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:67: TEST_recovery_scrub: for pg in $(seq 0 $(expr $PGS - 1)) 2019-12-21T15:47:27.890 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:69: TEST_recovery_scrub: printf %x 15 2019-12-21T15:47:27.891 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1911: run_in_background: pg_scrub 1.d 2019-12-21T15:47:27.893 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1911: run_in_background: sed 's/^/25813: /' 2019-12-21T15:47:27.894 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:69: TEST_recovery_scrub: run_in_background pids pg_scrub 1.f 2019-12-21T15:47:27.894 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1907: run_in_background: local pid_variable=pids 2019-12-21T15:47:27.894 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1908: run_in_background: shift 2019-12-21T15:47:27.895 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1912: run_in_background: eval 'pids+=" 29963"' 2019-12-21T15:47:27.895 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1912: run_in_background: pids+=' 29963' 2019-12-21T15:47:27.895 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:67: TEST_recovery_scrub: for pg in $(seq 0 $(expr $PGS - 1)) 2019-12-21T15:47:27.900 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1911: run_in_background: sed 's/^/25813: /' 2019-12-21T15:47:27.903 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:69: TEST_recovery_scrub: printf %x 16 2019-12-21T15:47:27.904 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:69: TEST_recovery_scrub: run_in_background pids pg_scrub 1.10 2019-12-21T15:47:27.905 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1907: run_in_background: local pid_variable=pids 2019-12-21T15:47:27.905 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1908: run_in_background: shift 2019-12-21T15:47:27.905 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1912: run_in_background: eval 'pids+=" 29967"' 2019-12-21T15:47:27.905 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1912: run_in_background: pids+=' 29967' 2019-12-21T15:47:27.905 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:67: TEST_recovery_scrub: for pg in $(seq 0 $(expr $PGS - 1)) 2019-12-21T15:47:27.909 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:69: TEST_recovery_scrub: printf %x 17 2019-12-21T15:47:27.911 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:69: TEST_recovery_scrub: run_in_background pids pg_scrub 1.11 2019-12-21T15:47:27.911 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1907: run_in_background: local pid_variable=pids 2019-12-21T15:47:27.911 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1908: run_in_background: shift 2019-12-21T15:47:27.911 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1912: run_in_background: eval 'pids+=" 29973"' 2019-12-21T15:47:27.911 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1912: run_in_background: pids+=' 29973' 2019-12-21T15:47:27.911 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:67: TEST_recovery_scrub: for pg in $(seq 0 $(expr $PGS - 1)) 2019-12-21T15:47:27.913 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1911: run_in_background: pg_scrub 1.f 2019-12-21T15:47:27.913 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1911: run_in_background: sed 's/^/25813: /' 2019-12-21T15:47:27.920 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1911: run_in_background: pg_scrub 1.e 2019-12-21T15:47:27.920 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1911: run_in_background: sed 's/^/25813: /' 2019-12-21T15:47:27.920 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:69: TEST_recovery_scrub: printf %x 18 2019-12-21T15:47:27.920 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:69: TEST_recovery_scrub: run_in_background pids pg_scrub 1.12 2019-12-21T15:47:27.921 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1907: run_in_background: local pid_variable=pids 2019-12-21T15:47:27.921 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1908: run_in_background: shift 2019-12-21T15:47:27.921 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1912: run_in_background: eval 'pids+=" 29979"' 2019-12-21T15:47:27.921 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1912: run_in_background: pids+=' 29979' 2019-12-21T15:47:27.921 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:67: TEST_recovery_scrub: for pg in $(seq 0 $(expr $PGS - 1)) 2019-12-21T15:47:27.929 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1911: run_in_background: pg_scrub 1.10 2019-12-21T15:47:27.930 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:69: TEST_recovery_scrub: printf %x 19 2019-12-21T15:47:27.931 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:69: TEST_recovery_scrub: run_in_background pids pg_scrub 1.13 2019-12-21T15:47:27.932 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1907: run_in_background: local pid_variable=pids 2019-12-21T15:47:27.932 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1908: run_in_background: shift 2019-12-21T15:47:27.932 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1912: run_in_background: eval 'pids+=" 29982"' 2019-12-21T15:47:27.932 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1912: run_in_background: pids+=' 29982' 2019-12-21T15:47:27.932 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:67: TEST_recovery_scrub: for pg in $(seq 0 $(expr $PGS - 1)) 2019-12-21T15:47:27.940 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:69: TEST_recovery_scrub: printf %x 20 2019-12-21T15:47:27.941 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:69: TEST_recovery_scrub: run_in_background pids pg_scrub 1.14 2019-12-21T15:47:27.941 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1907: run_in_background: local pid_variable=pids 2019-12-21T15:47:27.941 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1908: run_in_background: shift 2019-12-21T15:47:27.942 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1912: run_in_background: eval 'pids+=" 29990"' 2019-12-21T15:47:27.942 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1912: run_in_background: pids+=' 29990' 2019-12-21T15:47:27.942 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:67: TEST_recovery_scrub: for pg in $(seq 0 $(expr $PGS - 1)) 2019-12-21T15:47:27.942 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1911: run_in_background: pg_scrub 1.11 2019-12-21T15:47:27.943 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1911: run_in_background: sed 's/^/25813: /' 2019-12-21T15:47:27.943 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1911: run_in_background: sed 's/^/25813: /' 2019-12-21T15:47:27.944 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1911: run_in_background: pg_scrub 1.12 2019-12-21T15:47:27.953 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1911: run_in_background: pg_scrub 1.14 2019-12-21T15:47:27.955 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:69: TEST_recovery_scrub: printf %x 21 2019-12-21T15:47:27.961 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:69: TEST_recovery_scrub: run_in_background pids pg_scrub 1.15 2019-12-21T15:47:27.962 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1907: run_in_background: local pid_variable=pids 2019-12-21T15:47:27.962 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1908: run_in_background: shift 2019-12-21T15:47:27.969 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1912: run_in_background: eval 'pids+=" 30002"' 2019-12-21T15:47:27.969 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1912: run_in_background: pids+=' 30002' 2019-12-21T15:47:27.969 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:67: TEST_recovery_scrub: for pg in $(seq 0 $(expr $PGS - 1)) 2019-12-21T15:47:27.969 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1911: run_in_background: sed 's/^/25813: /' 2019-12-21T15:47:27.970 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1911: run_in_background: sed 's/^/25813: /' 2019-12-21T15:47:27.971 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:69: TEST_recovery_scrub: printf %x 22 2019-12-21T15:47:27.972 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:69: TEST_recovery_scrub: run_in_background pids pg_scrub 1.16 2019-12-21T15:47:27.972 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1907: run_in_background: local pid_variable=pids 2019-12-21T15:47:27.972 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1908: run_in_background: shift 2019-12-21T15:47:27.972 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1912: run_in_background: eval 'pids+=" 30012"' 2019-12-21T15:47:27.972 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1912: run_in_background: pids+=' 30012' 2019-12-21T15:47:27.972 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:67: TEST_recovery_scrub: for pg in $(seq 0 $(expr $PGS - 1)) 2019-12-21T15:47:27.973 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1911: run_in_background: pg_scrub 1.13 2019-12-21T15:47:27.980 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:69: TEST_recovery_scrub: printf %x 23 2019-12-21T15:47:27.981 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:69: TEST_recovery_scrub: run_in_background pids pg_scrub 1.17 2019-12-21T15:47:27.982 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1907: run_in_background: local pid_variable=pids 2019-12-21T15:47:27.982 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1908: run_in_background: shift 2019-12-21T15:47:27.982 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1912: run_in_background: eval 'pids+=" 30017"' 2019-12-21T15:47:27.982 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1912: run_in_background: pids+=' 30017' 2019-12-21T15:47:27.982 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:67: TEST_recovery_scrub: for pg in $(seq 0 $(expr $PGS - 1)) 2019-12-21T15:47:27.993 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:69: TEST_recovery_scrub: printf %x 24 2019-12-21T15:47:27.999 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:69: TEST_recovery_scrub: run_in_background pids pg_scrub 1.18 2019-12-21T15:47:27.999 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1907: run_in_background: local pid_variable=pids 2019-12-21T15:47:27.999 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1908: run_in_background: shift 2019-12-21T15:47:28.000 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1912: run_in_background: eval 'pids+=" 30024"' 2019-12-21T15:47:28.000 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1912: run_in_background: pids+=' 30024' 2019-12-21T15:47:28.000 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:67: TEST_recovery_scrub: for pg in $(seq 0 $(expr $PGS - 1)) 2019-12-21T15:47:28.000 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1911: run_in_background: pg_scrub 1.16 2019-12-21T15:47:28.000 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1911: run_in_background: sed 's/^/25813: /' 2019-12-21T15:47:28.001 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:69: TEST_recovery_scrub: printf %x 25 2019-12-21T15:47:28.002 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1911: run_in_background: pg_scrub 1.15 2019-12-21T15:47:28.005 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:69: TEST_recovery_scrub: run_in_background pids pg_scrub 1.19 2019-12-21T15:47:28.005 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1907: run_in_background: local pid_variable=pids 2019-12-21T15:47:28.005 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1908: run_in_background: shift 2019-12-21T15:47:28.005 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1912: run_in_background: eval 'pids+=" 30028"' 2019-12-21T15:47:28.005 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1912: run_in_background: pids+=' 30028' 2019-12-21T15:47:28.005 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:67: TEST_recovery_scrub: for pg in $(seq 0 $(expr $PGS - 1)) 2019-12-21T15:47:28.006 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1911: run_in_background: sed 's/^/25813: /' 2019-12-21T15:47:28.011 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:69: TEST_recovery_scrub: printf %x 26 2019-12-21T15:47:28.017 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:69: TEST_recovery_scrub: run_in_background pids pg_scrub 1.1a 2019-12-21T15:47:28.017 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1907: run_in_background: local pid_variable=pids 2019-12-21T15:47:28.017 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1908: run_in_background: shift 2019-12-21T15:47:28.018 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1912: run_in_background: eval 'pids+=" 30034"' 2019-12-21T15:47:28.018 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1912: run_in_background: pids+=' 30034' 2019-12-21T15:47:28.018 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:67: TEST_recovery_scrub: for pg in $(seq 0 $(expr $PGS - 1)) 2019-12-21T15:47:28.019 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1911: run_in_background: sed 's/^/25813: /' 2019-12-21T15:47:28.020 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1911: run_in_background: pg_scrub 1.17 2019-12-21T15:47:28.025 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1911: run_in_background: sed 's/^/25813: /' 2019-12-21T15:47:28.026 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1911: run_in_background: pg_scrub 1.18 2019-12-21T15:47:28.027 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1911: run_in_background: sed 's/^/25813: /' 2019-12-21T15:47:28.027 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1911: run_in_background: pg_scrub 1.19 2019-12-21T15:47:28.033 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:69: TEST_recovery_scrub: printf %x 27 2019-12-21T15:47:28.039 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:69: TEST_recovery_scrub: run_in_background pids pg_scrub 1.1b 2019-12-21T15:47:28.039 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1907: run_in_background: local pid_variable=pids 2019-12-21T15:47:28.039 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1908: run_in_background: shift 2019-12-21T15:47:28.039 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1912: run_in_background: eval 'pids+=" 30056"' 2019-12-21T15:47:28.039 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1912: run_in_background: pids+=' 30056' 2019-12-21T15:47:28.039 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:67: TEST_recovery_scrub: for pg in $(seq 0 $(expr $PGS - 1)) 2019-12-21T15:47:28.043 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1911: run_in_background: pg_scrub 1.1a 2019-12-21T15:47:28.046 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:69: TEST_recovery_scrub: printf %x 28 2019-12-21T15:47:28.047 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:69: TEST_recovery_scrub: run_in_background pids pg_scrub 1.1c 2019-12-21T15:47:28.047 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1907: run_in_background: local pid_variable=pids 2019-12-21T15:47:28.047 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1908: run_in_background: shift 2019-12-21T15:47:28.047 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1912: run_in_background: eval 'pids+=" 30065"' 2019-12-21T15:47:28.047 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1912: run_in_background: pids+=' 30065' 2019-12-21T15:47:28.047 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:67: TEST_recovery_scrub: for pg in $(seq 0 $(expr $PGS - 1)) 2019-12-21T15:47:28.050 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1911: run_in_background: pg_scrub 1.1b 2019-12-21T15:47:28.051 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1911: run_in_background: sed 's/^/25813: /' 2019-12-21T15:47:28.053 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1911: run_in_background: sed 's/^/25813: /' 2019-12-21T15:47:28.055 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:69: TEST_recovery_scrub: printf %x 29 2019-12-21T15:47:28.066 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:69: TEST_recovery_scrub: run_in_background pids pg_scrub 1.1d 2019-12-21T15:47:28.066 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1907: run_in_background: local pid_variable=pids 2019-12-21T15:47:28.066 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1908: run_in_background: shift 2019-12-21T15:47:28.066 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1912: run_in_background: eval 'pids+=" 30081"' 2019-12-21T15:47:28.066 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1912: run_in_background: pids+=' 30081' 2019-12-21T15:47:28.067 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:67: TEST_recovery_scrub: for pg in $(seq 0 $(expr $PGS - 1)) 2019-12-21T15:47:28.071 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1911: run_in_background: pg_scrub 1.1c 2019-12-21T15:47:28.075 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1911: run_in_background: sed 's/^/25813: /' 2019-12-21T15:47:28.082 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:69: TEST_recovery_scrub: printf %x 30 2019-12-21T15:47:28.083 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:69: TEST_recovery_scrub: run_in_background pids pg_scrub 1.1e 2019-12-21T15:47:28.084 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1907: run_in_background: local pid_variable=pids 2019-12-21T15:47:28.084 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1908: run_in_background: shift 2019-12-21T15:47:28.084 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1912: run_in_background: eval 'pids+=" 30088"' 2019-12-21T15:47:28.084 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1911: run_in_background: pg_scrub 1.1d 2019-12-21T15:47:28.085 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1912: run_in_background: pids+=' 30088' 2019-12-21T15:47:28.085 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:67: TEST_recovery_scrub: for pg in $(seq 0 $(expr $PGS - 1)) 2019-12-21T15:47:28.085 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1911: run_in_background: sed 's/^/25813: /' 2019-12-21T15:47:28.093 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1911: run_in_background: pg_scrub 1.1e 2019-12-21T15:47:28.098 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:69: TEST_recovery_scrub: printf %x 31 2019-12-21T15:47:28.107 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1911: run_in_background: sed 's/^/25813: /' 2019-12-21T15:47:28.115 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:69: TEST_recovery_scrub: run_in_background pids pg_scrub 1.1f 2019-12-21T15:47:28.116 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1907: run_in_background: local pid_variable=pids 2019-12-21T15:47:28.116 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1908: run_in_background: shift 2019-12-21T15:47:28.116 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1912: run_in_background: eval 'pids+=" 30121"' 2019-12-21T15:47:28.116 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1912: run_in_background: pids+=' 30121' 2019-12-21T15:47:28.116 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:71: TEST_recovery_scrub: ceph pg dump pgs 2019-12-21T15:47:28.127 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1911: run_in_background: pg_scrub 1.1f 2019-12-21T15:47:28.128 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1911: run_in_background: sed 's/^/25813: /' 2019-12-21T15:47:30.598 INFO:tasks.workunit.client.0.smithi172.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2019-12-21T15:47:30.598 INFO:tasks.workunit.client.0.smithi172.stdout:1.1f 1 0 0 0 0 52428800 0 0 13 13 active+clean 2019-12-21T15:47:16.961194+0000 47'13 47:23 [5] 5 [5] 5 0'0 2019-12-21T15:47:15.630468+0000 0'0 2019-12-21T15:47:15.630468+0000 0 2019-12-21T15:47:30.599 INFO:tasks.workunit.client.0.smithi172.stdout:1.1e 0 0 0 0 0 0 0 0 0 0 activating 2019-12-21T15:47:28.501680+0000 0'0 50:14 [3,0,7,4] 3 [3,0,7,4] 3 0'0 2019-12-21T15:47:15.630468+0000 0'0 2019-12-21T15:47:15.630468+0000 0 2019-12-21T15:47:30.599 INFO:tasks.workunit.client.0.smithi172.stdout:1.1d 0 0 0 0 0 0 0 0 0 0 active+clean 2019-12-21T15:47:15.914852+0000 0'0 45:10 [1] 1 [1] 1 0'0 2019-12-21T15:47:15.630468+0000 0'0 2019-12-21T15:47:15.630468+0000 0 2019-12-21T15:47:30.599 INFO:tasks.workunit.client.0.smithi172.stdout:1.1c 0 0 0 0 0 0 0 0 0 0 active+clean 2019-12-21T15:47:28.593393+0000 0'0 50:23 [0,3,4,1] 0 [0,3,4,1] 0 0'0 2019-12-21T15:47:15.630468+0000 0'0 2019-12-21T15:47:15.630468+0000 0 2019-12-21T15:47:30.599 INFO:tasks.workunit.client.0.smithi172.stdout:1.1b 0 0 0 0 0 0 0 0 0 0 activating 2019-12-21T15:47:28.393951+0000 0'0 50:18 [0,1,2,7] 0 [0,1,2,7] 0 0'0 2019-12-21T15:47:15.630468+0000 0'0 2019-12-21T15:47:15.630468+0000 0 2019-12-21T15:47:30.599 INFO:tasks.workunit.client.0.smithi172.stdout:1.1a 0 0 0 0 0 0 0 0 0 0 active+clean 2019-12-21T15:47:15.915082+0000 0'0 45:10 [1] 1 [1] 1 0'0 2019-12-21T15:47:15.630468+0000 0'0 2019-12-21T15:47:15.630468+0000 0 2019-12-21T15:47:30.599 INFO:tasks.workunit.client.0.smithi172.stdout:1.19 0 0 0 0 0 0 0 0 0 0 activating 2019-12-21T15:47:28.501182+0000 0'0 50:14 [3,0,4,2] 3 [3,0,4,2] 3 0'0 2019-12-21T15:47:15.630468+0000 0'0 2019-12-21T15:47:15.630468+0000 0 2019-12-21T15:47:30.600 INFO:tasks.workunit.client.0.smithi172.stdout:1.18 0 0 0 0 0 0 0 0 0 0 active+clean 2019-12-21T15:47:16.960727+0000 0'0 46:10 [5] 5 [5] 5 0'0 2019-12-21T15:47:15.630468+0000 0'0 2019-12-21T15:47:15.630468+0000 0 2019-12-21T15:47:30.600 INFO:tasks.workunit.client.0.smithi172.stdout:1.17 0 0 0 0 0 0 0 0 0 0 active+clean 2019-12-21T15:47:28.597790+0000 0'0 50:23 [0,4,5,6] 0 [0,4,5,6] 0 0'0 2019-12-21T15:47:15.630468+0000 0'0 2019-12-21T15:47:15.630468+0000 0 2019-12-21T15:47:30.600 INFO:tasks.workunit.client.0.smithi172.stdout:1.16 0 0 0 0 0 0 0 0 0 0 activating 2019-12-21T15:47:28.400640+0000 0'0 50:13 [6,0,7,4] 6 [6,0,7,4] 6 0'0 2019-12-21T15:47:15.630468+0000 0'0 2019-12-21T15:47:15.630468+0000 0 2019-12-21T15:47:30.600 INFO:tasks.workunit.client.0.smithi172.stdout:1.15 0 0 0 0 0 0 0 0 0 0 active+clean 2019-12-21T15:47:15.914943+0000 0'0 45:10 [1] 1 [1] 1 0'0 2019-12-21T15:47:15.630468+0000 0'0 2019-12-21T15:47:15.630468+0000 0 2019-12-21T15:47:30.600 INFO:tasks.workunit.client.0.smithi172.stdout:1.14 0 0 0 0 0 0 0 0 0 0 active+clean 2019-12-21T15:47:16.516648+0000 0'0 45:10 [2] 2 [2] 2 0'0 2019-12-21T15:47:15.630468+0000 0'0 2019-12-21T15:47:15.630468+0000 0 2019-12-21T15:47:30.600 INFO:tasks.workunit.client.0.smithi172.stdout:1.13 0 0 0 0 0 0 0 0 0 0 active+clean 2019-12-21T15:47:17.051872+0000 0'0 46:11 [4] 4 [4] 4 0'0 2019-12-21T15:47:15.630468+0000 0'0 2019-12-21T15:47:15.630468+0000 0 2019-12-21T15:47:30.601 INFO:tasks.workunit.client.0.smithi172.stdout:1.12 0 0 0 0 0 0 0 0 0 0 activating 2019-12-21T15:47:28.394847+0000 0'0 50:18 [0,1,6,7] 0 [0,1,6,7] 0 0'0 2019-12-21T15:47:15.630468+0000 0'0 2019-12-21T15:47:15.630468+0000 0 2019-12-21T15:47:30.601 INFO:tasks.workunit.client.0.smithi172.stdout:1.11 1 0 3 0 0 52428800 0 0 13 13 active+recovering+degraded 2019-12-21T15:47:28.652533+0000 47'13 50:36 [0,1,3,4] 0 [0,1,3,4] 0 0'0 2019-12-21T15:47:15.630468+0000 0'0 2019-12-21T15:47:15.630468+0000 0 2019-12-21T15:47:30.601 INFO:tasks.workunit.client.0.smithi172.stdout:1.10 1 0 0 0 0 12582912 0 0 3 3 active+clean 2019-12-21T15:47:16.960725+0000 47'3 47:13 [5] 5 [5] 5 0'0 2019-12-21T15:47:15.630468+0000 0'0 2019-12-21T15:47:15.630468+0000 0 2019-12-21T15:47:30.601 INFO:tasks.workunit.client.0.smithi172.stdout:1.5 0 0 0 0 0 0 0 0 0 0 active+clean 2019-12-21T15:47:17.051902+0000 0'0 46:11 [4] 4 [4] 4 0'0 2019-12-21T15:47:15.630468+0000 0'0 2019-12-21T15:47:15.630468+0000 0 2019-12-21T15:47:30.601 INFO:tasks.workunit.client.0.smithi172.stdout:1.4 0 0 0 0 0 0 0 0 0 0 activating 2019-12-21T15:47:28.505610+0000 0'0 50:14 [3,0,6,1] 3 [3,0,6,1] 3 0'0 2019-12-21T15:47:15.630468+0000 0'0 2019-12-21T15:47:15.630468+0000 0 2019-12-21T15:47:30.601 INFO:tasks.workunit.client.0.smithi172.stdout:1.3 0 0 0 0 0 0 0 0 0 0 active+clean 2019-12-21T15:47:17.068415+0000 0'0 46:11 [4] 4 [4] 4 0'0 2019-12-21T15:47:15.630468+0000 0'0 2019-12-21T15:47:15.630468+0000 0 2019-12-21T15:47:30.602 INFO:tasks.workunit.client.0.smithi172.stdout:1.2 0 0 0 0 0 0 0 0 0 0 activating 2019-12-21T15:47:28.394400+0000 0'0 50:18 [0,7,1,3] 0 [0,7,1,3] 0 0'0 2019-12-21T15:47:15.630468+0000 0'0 2019-12-21T15:47:15.630468+0000 0 2019-12-21T15:47:30.602 INFO:tasks.workunit.client.0.smithi172.stdout:1.0 0 0 0 0 0 0 0 0 0 0 active+clean 2019-12-21T15:47:16.172216+0000 0'0 45:10 [7] 7 [7] 7 0'0 2019-12-21T15:47:15.630468+0000 0'0 2019-12-21T15:47:15.630468+0000 0 2019-12-21T15:47:30.602 INFO:tasks.workunit.client.0.smithi172.stdout:1.1 0 0 0 0 0 0 0 0 0 0 active+clean 2019-12-21T15:47:16.961106+0000 0'0 46:10 [5] 5 [5] 5 0'0 2019-12-21T15:47:15.630468+0000 0'0 2019-12-21T15:47:15.630468+0000 0 2019-12-21T15:47:30.602 INFO:tasks.workunit.client.0.smithi172.stdout:1.6 0 0 0 0 0 0 0 0 0 0 activating 2019-12-21T15:47:28.399405+0000 0'0 50:13 [6,7,3,4] 6 [6,7,3,4] 6 0'0 2019-12-21T15:47:15.630468+0000 0'0 2019-12-21T15:47:15.630468+0000 0 2019-12-21T15:47:30.602 INFO:tasks.workunit.client.0.smithi172.stdout:1.7 0 0 0 0 0 0 0 0 0 0 activating 2019-12-21T15:47:28.399968+0000 0'0 50:13 [6,7,3,2] 6 [6,7,3,2] 6 0'0 2019-12-21T15:47:15.630468+0000 0'0 2019-12-21T15:47:15.630468+0000 0 2019-12-21T15:47:30.602 INFO:tasks.workunit.client.0.smithi172.stdout:1.8 1 0 0 0 0 52428800 0 0 13 13 active+clean 2019-12-21T15:47:16.926299+0000 47'13 47:23 [5] 5 [5] 5 0'0 2019-12-21T15:47:15.630468+0000 0'0 2019-12-21T15:47:15.630468+0000 0 2019-12-21T15:47:30.602 INFO:tasks.workunit.client.0.smithi172.stdout:1.9 0 0 0 0 0 0 0 0 0 0 active+clean 2019-12-21T15:47:15.914771+0000 0'0 45:10 [1] 1 [1] 1 0'0 2019-12-21T15:47:15.630468+0000 0'0 2019-12-21T15:47:15.630468+0000 0 2019-12-21T15:47:30.603 INFO:tasks.workunit.client.0.smithi172.stdout:1.a 0 0 0 0 0 0 0 0 0 0 active+clean 2019-12-21T15:47:15.898265+0000 0'0 45:10 [1] 1 [1] 1 0'0 2019-12-21T15:47:15.630468+0000 0'0 2019-12-21T15:47:15.630468+0000 0 2019-12-21T15:47:30.603 INFO:tasks.workunit.client.0.smithi172.stdout:1.b 0 0 0 0 0 0 0 0 0 0 active+clean 2019-12-21T15:47:15.915218+0000 0'0 45:10 [1] 1 [1] 1 0'0 2019-12-21T15:47:15.630468+0000 0'0 2019-12-21T15:47:15.630468+0000 0 2019-12-21T15:47:30.603 INFO:tasks.workunit.client.0.smithi172.stdout:1.c 0 0 0 0 0 0 0 0 0 0 activating 2019-12-21T15:47:28.521500+0000 0'0 50:14 [3,0,4,6] 3 [3,0,4,6] 3 0'0 2019-12-21T15:47:15.630468+0000 0'0 2019-12-21T15:47:15.630468+0000 0 2019-12-21T15:47:30.603 INFO:tasks.workunit.client.0.smithi172.stdout:1.d 0 0 0 0 0 0 0 0 0 0 active+clean 2019-12-21T15:47:16.172247+0000 0'0 45:10 [7] 7 [7] 7 0'0 2019-12-21T15:47:15.630468+0000 0'0 2019-12-21T15:47:15.630468+0000 0 2019-12-21T15:47:30.603 INFO:tasks.workunit.client.0.smithi172.stdout:1.e 0 0 0 0 0 0 0 0 0 0 active+clean 2019-12-21T15:47:15.914345+0000 0'0 45:10 [1] 1 [1] 1 0'0 2019-12-21T15:47:15.630468+0000 0'0 2019-12-21T15:47:15.630468+0000 0 2019-12-21T15:47:30.603 INFO:tasks.workunit.client.0.smithi172.stdout:1.f 0 0 0 0 0 0 0 0 0 0 activating 2019-12-21T15:47:28.401561+0000 0'0 50:13 [6,0,3,4] 6 [6,0,3,4] 6 0'0 2019-12-21T15:47:15.630468+0000 0'0 2019-12-21T15:47:15.630468+0000 0 2019-12-21T15:47:30.604 INFO:tasks.workunit.client.0.smithi172.stdout: 2019-12-21T15:47:30.604 INFO:tasks.workunit.client.0.smithi172.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilisation. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2019-12-21T15:47:30.624 INFO:tasks.workunit.client.0.smithi172.stderr:dumped pgs 2019-12-21T15:47:30.642 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:72: TEST_recovery_scrub: wait_background pids 2019-12-21T15:47:30.642 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1943: wait_background: pids=' 29863 29870 29877 29884 29891 29898 29902 29910 29912 29918 29922 29924 29941 29949 29956 29963 29967 29973 29979 29982 29990 30002 30012 30017 30024 30028 30034 30056 30065 30081 30088 30121' 2019-12-21T15:47:30.642 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1945: wait_background: return_code=0 2019-12-21T15:47:30.643 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1946: wait_background: for pid in $pids 2019-12-21T15:47:30.643 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1947: wait_background: wait 29863 2019-12-21T15:47:35.071 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1697: pg_scrub: local pgid=1.5 2019-12-21T15:47:35.072 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1698: pg_scrub: get_last_scrub_stamp 1.5 2019-12-21T15:47:35.072 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.5 2019-12-21T15:47:35.072 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:35.072 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:35.072 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.5") | .last_scrub_stamp' 2019-12-21T15:47:35.073 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1698: pg_scrub: local last_scrub=2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:35.073 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1699: pg_scrub: ceph pg scrub 1.5 2019-12-21T15:47:35.073 INFO:tasks.workunit.client.0.smithi172.stderr:25813: instructing pg 1.5 on osd.4 to scrub 2019-12-21T15:47:35.073 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1700: pg_scrub: wait_for_scrub 1.5 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:35.073 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1789: wait_for_scrub: local pgid=1.5 2019-12-21T15:47:35.073 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1790: wait_for_scrub: local last_scrub=2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:35.073 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1791: wait_for_scrub: local sname=last_scrub_stamp 2019-12-21T15:47:35.074 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i=0 )) 2019-12-21T15:47:35.074 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:35.074 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.5 last_scrub_stamp 2019-12-21T15:47:35.074 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.5 2019-12-21T15:47:35.074 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:35.074 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:35.074 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.5") | .last_scrub_stamp' 2019-12-21T15:47:35.075 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:35.075 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:35.075 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:35.075 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:35.075 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.5 last_scrub_stamp 2019-12-21T15:47:35.075 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.5 2019-12-21T15:47:35.075 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:35.076 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:35.076 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.5") | .last_scrub_stamp' 2019-12-21T15:47:35.076 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:35.076 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:35.076 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:35.181 INFO:tasks.workunit.client.0.smithi172.stderr:25813: 25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1697: pg_scrub: local pgid=1.0 2019-12-21T15:47:35.181 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1698: pg_scrub: get_last_scrub_stamp 1.0 2019-12-21T15:47:35.181 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.0 2019-12-21T15:47:35.181 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:35.181 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:35.181 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2019-12-21T15:47:35.182 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1698: pg_scrub: local last_scrub=2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:35.182 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1699: pg_scrub: ceph pg scrub 1.0 2019-12-21T15:47:35.182 INFO:tasks.workunit.client.0.smithi172.stderr:25813: instructing pg 1.0 on osd.7 to scrub 2019-12-21T15:47:35.182 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1700: pg_scrub: wait_for_scrub 1.0 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:35.182 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1789: wait_for_scrub: local pgid=1.0 2019-12-21T15:47:35.182 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1790: wait_for_scrub: local last_scrub=2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:35.182 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1791: wait_for_scrub: local sname=last_scrub_stamp 2019-12-21T15:47:35.183 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i=0 )) 2019-12-21T15:47:35.183 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:35.183 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2019-12-21T15:47:35.183 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.0 2019-12-21T15:47:35.183 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:35.183 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2019-12-21T15:47:35.184 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:35.184 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:35.184 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:35.184 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:35.184 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:35.184 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2019-12-21T15:47:35.185 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.0 2019-12-21T15:47:35.185 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:35.185 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:35.185 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2019-12-21T15:47:35.185 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:35.185 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:35.186 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:35.199 INFO:tasks.workunit.client.0.smithi172.stderr:25813: 25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1697: pg_scrub: local pgid=1.2 2019-12-21T15:47:35.199 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1698: pg_scrub: get_last_scrub_stamp 1.2 2019-12-21T15:47:35.199 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.2 2019-12-21T15:47:35.199 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:35.200 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:35.200 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.2") | .last_scrub_stamp' 2019-12-21T15:47:35.200 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1698: pg_scrub: local last_scrub=2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:35.200 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1699: pg_scrub: ceph pg scrub 1.2 2019-12-21T15:47:35.200 INFO:tasks.workunit.client.0.smithi172.stderr:25813: instructing pg 1.2 on osd.0 to scrub 2019-12-21T15:47:35.200 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1700: pg_scrub: wait_for_scrub 1.2 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:35.200 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1789: wait_for_scrub: local pgid=1.2 2019-12-21T15:47:35.201 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1790: wait_for_scrub: local last_scrub=2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:35.202 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1791: wait_for_scrub: local sname=last_scrub_stamp 2019-12-21T15:47:35.202 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i=0 )) 2019-12-21T15:47:35.202 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:35.202 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.2 last_scrub_stamp 2019-12-21T15:47:35.203 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.2 2019-12-21T15:47:35.203 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:35.203 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.2") | .last_scrub_stamp' 2019-12-21T15:47:35.203 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:35.203 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:35.203 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:35.203 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:35.204 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:35.204 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.2 last_scrub_stamp 2019-12-21T15:47:35.204 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.2 2019-12-21T15:47:35.204 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:35.204 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:35.205 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.2") | .last_scrub_stamp' 2019-12-21T15:47:35.205 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:35.205 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:35.205 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:35.637 INFO:tasks.workunit.client.0.smithi172.stderr:25813: 25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1697: pg_scrub: local pgid=1.b 2019-12-21T15:47:35.638 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1698: pg_scrub: get_last_scrub_stamp 1.b 2019-12-21T15:47:35.638 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.b 2019-12-21T15:47:35.638 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:35.638 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:35.638 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.b") | .last_scrub_stamp' 2019-12-21T15:47:35.639 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1698: pg_scrub: local last_scrub=2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:35.639 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1699: pg_scrub: ceph pg scrub 1.b 2019-12-21T15:47:35.639 INFO:tasks.workunit.client.0.smithi172.stderr:25813: instructing pg 1.b on osd.1 to scrub 2019-12-21T15:47:35.639 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1700: pg_scrub: wait_for_scrub 1.b 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:35.639 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1789: wait_for_scrub: local pgid=1.b 2019-12-21T15:47:35.639 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1790: wait_for_scrub: local last_scrub=2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:35.639 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1791: wait_for_scrub: local sname=last_scrub_stamp 2019-12-21T15:47:35.640 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i=0 )) 2019-12-21T15:47:35.640 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:35.640 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.b last_scrub_stamp 2019-12-21T15:47:35.640 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.b 2019-12-21T15:47:35.640 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:35.640 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.b") | .last_scrub_stamp' 2019-12-21T15:47:35.640 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:35.641 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:35.641 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:35.641 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:35.641 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:35.641 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.b last_scrub_stamp 2019-12-21T15:47:35.641 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.b 2019-12-21T15:47:35.641 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:35.642 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.b") | .last_scrub_stamp' 2019-12-21T15:47:35.642 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:35.642 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:35.642 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:35.642 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:35.807 INFO:tasks.workunit.client.0.smithi172.stderr:25813: 25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1697: pg_scrub: local pgid=1.9 2019-12-21T15:47:35.807 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1698: pg_scrub: get_last_scrub_stamp 1.9 2019-12-21T15:47:35.807 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.9 2019-12-21T15:47:35.807 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:35.807 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.9") | .last_scrub_stamp' 2019-12-21T15:47:35.808 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:35.808 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1698: pg_scrub: local last_scrub=2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:35.808 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1699: pg_scrub: ceph pg scrub 1.9 2019-12-21T15:47:35.808 INFO:tasks.workunit.client.0.smithi172.stderr:25813: instructing pg 1.9 on osd.1 to scrub 2019-12-21T15:47:35.808 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1700: pg_scrub: wait_for_scrub 1.9 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:35.808 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1789: wait_for_scrub: local pgid=1.9 2019-12-21T15:47:35.809 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1790: wait_for_scrub: local last_scrub=2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:35.809 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1791: wait_for_scrub: local sname=last_scrub_stamp 2019-12-21T15:47:35.809 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i=0 )) 2019-12-21T15:47:35.809 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:35.809 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.9 last_scrub_stamp 2019-12-21T15:47:35.809 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.9 2019-12-21T15:47:35.809 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:35.810 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.9") | .last_scrub_stamp' 2019-12-21T15:47:35.810 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:35.810 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:35.810 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:35.810 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:35.811 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:35.811 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.9 last_scrub_stamp 2019-12-21T15:47:35.811 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.9 2019-12-21T15:47:35.811 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:35.811 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:35.812 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.9") | .last_scrub_stamp' 2019-12-21T15:47:35.812 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:35.812 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:35.812 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:35.940 INFO:tasks.workunit.client.0.smithi172.stderr:25813: 25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1697: pg_scrub: local pgid=1.e 2019-12-21T15:47:35.940 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1698: pg_scrub: get_last_scrub_stamp 1.e 2019-12-21T15:47:35.940 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.e 2019-12-21T15:47:35.941 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:35.941 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:35.941 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.e") | .last_scrub_stamp' 2019-12-21T15:47:35.941 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1698: pg_scrub: local last_scrub=2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:35.941 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1699: pg_scrub: ceph pg scrub 1.e 2019-12-21T15:47:35.941 INFO:tasks.workunit.client.0.smithi172.stderr:25813: instructing pg 1.e on osd.1 to scrub 2019-12-21T15:47:35.941 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1700: pg_scrub: wait_for_scrub 1.e 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:35.942 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1789: wait_for_scrub: local pgid=1.e 2019-12-21T15:47:35.942 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1790: wait_for_scrub: local last_scrub=2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:35.942 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1791: wait_for_scrub: local sname=last_scrub_stamp 2019-12-21T15:47:35.942 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i=0 )) 2019-12-21T15:47:35.942 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:35.942 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.e last_scrub_stamp 2019-12-21T15:47:35.942 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.e 2019-12-21T15:47:35.943 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:35.943 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.e") | .last_scrub_stamp' 2019-12-21T15:47:35.943 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:35.943 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:35.943 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:35.943 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:35.943 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:35.944 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.e last_scrub_stamp 2019-12-21T15:47:35.944 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.e 2019-12-21T15:47:35.944 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:35.944 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:35.944 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.e") | .last_scrub_stamp' 2019-12-21T15:47:35.944 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:35.944 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:35.945 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:35.999 INFO:tasks.workunit.client.0.smithi172.stderr:25813: 25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1697: pg_scrub: local pgid=1.6 2019-12-21T15:47:35.999 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1698: pg_scrub: get_last_scrub_stamp 1.6 2019-12-21T15:47:35.999 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.6 2019-12-21T15:47:35.999 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:35.999 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:35.999 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.6") | .last_scrub_stamp' 2019-12-21T15:47:36.000 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1698: pg_scrub: local last_scrub=2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:36.000 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1699: pg_scrub: ceph pg scrub 1.6 2019-12-21T15:47:36.000 INFO:tasks.workunit.client.0.smithi172.stderr:25813: instructing pg 1.6 on osd.6 to scrub 2019-12-21T15:47:36.000 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1700: pg_scrub: wait_for_scrub 1.6 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:36.000 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1789: wait_for_scrub: local pgid=1.6 2019-12-21T15:47:36.000 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1790: wait_for_scrub: local last_scrub=2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:36.000 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1791: wait_for_scrub: local sname=last_scrub_stamp 2019-12-21T15:47:36.001 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i=0 )) 2019-12-21T15:47:36.001 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:36.001 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.6 last_scrub_stamp 2019-12-21T15:47:36.001 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.6 2019-12-21T15:47:36.001 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:36.001 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.6") | .last_scrub_stamp' 2019-12-21T15:47:36.001 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:36.002 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:36.002 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:36.002 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:36.002 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:36.002 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.6 last_scrub_stamp 2019-12-21T15:47:36.002 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.6 2019-12-21T15:47:36.002 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:36.003 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:36.003 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.6") | .last_scrub_stamp' 2019-12-21T15:47:36.003 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:36.003 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:36.003 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:36.097 INFO:tasks.workunit.client.0.smithi172.stderr:25813: 25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1697: pg_scrub: local pgid=1.13 2019-12-21T15:47:36.097 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1698: pg_scrub: get_last_scrub_stamp 1.13 2019-12-21T15:47:36.097 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.13 2019-12-21T15:47:36.097 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:36.098 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.13") | .last_scrub_stamp' 2019-12-21T15:47:36.098 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:36.098 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1698: pg_scrub: local last_scrub=2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:36.098 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1699: pg_scrub: ceph pg scrub 1.13 2019-12-21T15:47:36.098 INFO:tasks.workunit.client.0.smithi172.stderr:25813: instructing pg 1.13 on osd.4 to scrub 2019-12-21T15:47:36.098 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1700: pg_scrub: wait_for_scrub 1.13 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:36.098 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1789: wait_for_scrub: local pgid=1.13 2019-12-21T15:47:36.099 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1790: wait_for_scrub: local last_scrub=2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:36.099 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1791: wait_for_scrub: local sname=last_scrub_stamp 2019-12-21T15:47:36.099 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i=0 )) 2019-12-21T15:47:36.099 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:36.099 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.13 last_scrub_stamp 2019-12-21T15:47:36.099 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.13 2019-12-21T15:47:36.099 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:36.100 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.13") | .last_scrub_stamp' 2019-12-21T15:47:36.100 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:36.100 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:36.100 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:36.100 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:36.100 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:36.100 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.13 last_scrub_stamp 2019-12-21T15:47:36.101 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.13 2019-12-21T15:47:36.101 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:36.101 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:36.101 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.13") | .last_scrub_stamp' 2019-12-21T15:47:36.101 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:36.101 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:36.136 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( 25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1697: pg_scrub: local pgid=1.3 2019-12-21T15:47:36.136 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1698: pg_scrub: get_last_scrub_stamp 1.3 2019-12-21T15:47:36.136 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.3 2019-12-21T15:47:36.137 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:36.137 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:36.137 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2019-12-21T15:47:36.137 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1698: pg_scrub: local last_scrub=2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:36.137 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1699: pg_scrub: ceph pg scrub 1.3 2019-12-21T15:47:36.137 INFO:tasks.workunit.client.0.smithi172.stderr:25813: instructing pg 1.3 on osd.4 to scrub 2019-12-21T15:47:36.137 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1700: pg_scrub: wait_for_scrub 1.3 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:36.138 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1789: wait_for_scrub: local pgid=1.3 2019-12-21T15:47:36.138 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1790: wait_for_scrub: local last_scrub=2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:36.138 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1791: wait_for_scrub: local sname=last_scrub_stamp 2019-12-21T15:47:36.138 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i=0 )) 2019-12-21T15:47:36.138 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:36.138 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2019-12-21T15:47:36.138 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.3 2019-12-21T15:47:36.139 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:36.139 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2019-12-21T15:47:36.139 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:36.139 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:36.139 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:36.139 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:36.139 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:36.140 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2019-12-21T15:47:36.140 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.3 2019-12-21T15:47:36.140 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:36.140 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:36.140 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2019-12-21T15:47:36.140 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:36.140 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:36.141 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:36.147 INFO:tasks.workunit.client.0.smithi172.stderr:25813: 25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1697: pg_scrub: local pgid=1.4 2019-12-21T15:47:36.148 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1698: pg_scrub: get_last_scrub_stamp 1.4 2019-12-21T15:47:36.148 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.4 2019-12-21T15:47:36.148 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:36.148 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:36.148 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.4") | .last_scrub_stamp' 2019-12-21T15:47:36.148 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1698: pg_scrub: local last_scrub=2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:36.148 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1699: pg_scrub: ceph pg scrub 1.4 2019-12-21T15:47:36.149 INFO:tasks.workunit.client.0.smithi172.stderr:25813: instructing pg 1.4 on osd.3 to scrub 2019-12-21T15:47:36.149 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1700: pg_scrub: wait_for_scrub 1.4 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:36.149 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1789: wait_for_scrub: local pgid=1.4 2019-12-21T15:47:36.149 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1790: wait_for_scrub: local last_scrub=2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:36.149 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1791: wait_for_scrub: local sname=last_scrub_stamp 2019-12-21T15:47:36.149 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i=0 )) 2019-12-21T15:47:36.149 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:36.150 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.4 last_scrub_stamp 2019-12-21T15:47:36.150 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.4 2019-12-21T15:47:36.150 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:36.150 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:36.150 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.4") | .last_scrub_stamp' 2019-12-21T15:47:36.150 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:36.150 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:36.151 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:36.151 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:36.151 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.4 last_scrub_stamp 2019-12-21T15:47:36.151 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.4 2019-12-21T15:47:36.151 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:36.151 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:36.151 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.4") | .last_scrub_stamp' 2019-12-21T15:47:36.152 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:36.152 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:36.152 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:36.155 INFO:tasks.workunit.client.0.smithi172.stderr:25813: 25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1697: pg_scrub: local pgid=1.8 2019-12-21T15:47:36.156 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1698: pg_scrub: get_last_scrub_stamp 1.8 2019-12-21T15:47:36.156 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.8 2019-12-21T15:47:36.156 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:36.156 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.8") | .last_scrub_stamp' 2019-12-21T15:47:36.156 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:36.156 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1698: pg_scrub: local last_scrub=2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:36.157 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1699: pg_scrub: ceph pg scrub 1.8 2019-12-21T15:47:36.157 INFO:tasks.workunit.client.0.smithi172.stderr:25813: instructing pg 1.8 on osd.5 to scrub 2019-12-21T15:47:36.157 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1700: pg_scrub: wait_for_scrub 1.8 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:36.157 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1789: wait_for_scrub: local pgid=1.8 2019-12-21T15:47:36.157 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1790: wait_for_scrub: local last_scrub=2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:36.157 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1791: wait_for_scrub: local sname=last_scrub_stamp 2019-12-21T15:47:36.157 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i=0 )) 2019-12-21T15:47:36.158 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:36.158 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.8 last_scrub_stamp 2019-12-21T15:47:36.158 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.8 2019-12-21T15:47:36.158 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:36.158 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:36.158 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.8") | .last_scrub_stamp' 2019-12-21T15:47:36.158 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:36.159 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:36.159 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:36.159 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:36.159 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.8 last_scrub_stamp 2019-12-21T15:47:36.159 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.8 2019-12-21T15:47:36.159 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:36.159 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:36.160 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.8") | .last_scrub_stamp' 2019-12-21T15:47:36.160 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:36.160 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:36.160 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:36.164 INFO:tasks.workunit.client.0.smithi172.stderr:25813: 25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1697: pg_scrub: local pgid=1.1 2019-12-21T15:47:36.164 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1698: pg_scrub: get_last_scrub_stamp 1.1 2019-12-21T15:47:36.164 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.1 2019-12-21T15:47:36.164 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:36.164 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:36.165 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1") | .last_scrub_stamp' 2019-12-21T15:47:36.165 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1698: pg_scrub: local last_scrub=2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:36.165 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1699: pg_scrub: ceph pg scrub 1.1 2019-12-21T15:47:36.165 INFO:tasks.workunit.client.0.smithi172.stderr:25813: instructing pg 1.1 on osd.5 to scrub 2019-12-21T15:47:36.165 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1700: pg_scrub: wait_for_scrub 1.1 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:36.166 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1789: wait_for_scrub: local pgid=1.1 2019-12-21T15:47:36.166 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1790: wait_for_scrub: local last_scrub=2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:36.166 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1791: wait_for_scrub: local sname=last_scrub_stamp 2019-12-21T15:47:36.166 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i=0 )) 2019-12-21T15:47:36.166 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:36.166 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.1 last_scrub_stamp 2019-12-21T15:47:36.166 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.1 2019-12-21T15:47:36.167 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:36.167 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1") | .last_scrub_stamp' 2019-12-21T15:47:36.167 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:36.167 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:36.167 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:36.167 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:36.167 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:36.168 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.1 last_scrub_stamp 2019-12-21T15:47:36.168 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.1 2019-12-21T15:47:36.168 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:36.168 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:36.168 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1") | .last_scrub_stamp' 2019-12-21T15:47:36.168 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:36.168 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:36.169 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:36.269 INFO:tasks.workunit.client.0.smithi172.stderr:25813: 25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1697: pg_scrub: local pgid=1.f 2019-12-21T15:47:36.269 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1698: pg_scrub: get_last_scrub_stamp 1.f 2019-12-21T15:47:36.269 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.f 2019-12-21T15:47:36.269 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:36.269 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:36.269 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.f") | .last_scrub_stamp' 2019-12-21T15:47:36.270 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1698: pg_scrub: local last_scrub=2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:36.270 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1699: pg_scrub: ceph pg scrub 1.f 2019-12-21T15:47:36.270 INFO:tasks.workunit.client.0.smithi172.stderr:25813: instructing pg 1.f on osd.6 to scrub 2019-12-21T15:47:36.270 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1700: pg_scrub: wait_for_scrub 1.f 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:36.270 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1789: wait_for_scrub: local pgid=1.f 2019-12-21T15:47:36.270 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1790: wait_for_scrub: local last_scrub=2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:36.270 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1791: wait_for_scrub: local sname=last_scrub_stamp 2019-12-21T15:47:36.271 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i=0 )) 2019-12-21T15:47:36.271 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:36.271 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.f last_scrub_stamp 2019-12-21T15:47:36.271 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.f 2019-12-21T15:47:36.271 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:36.271 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:36.272 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.f") | .last_scrub_stamp' 2019-12-21T15:47:36.272 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:36.272 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:36.272 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:36.272 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:36.272 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.f last_scrub_stamp 2019-12-21T15:47:36.272 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.f 2019-12-21T15:47:36.273 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:36.273 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.f") | .last_scrub_stamp' 2019-12-21T15:47:36.273 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:36.273 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:36.273 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:36.273 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:36.406 INFO:tasks.workunit.client.0.smithi172.stderr:25813: 25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1697: pg_scrub: local pgid=1.d 2019-12-21T15:47:36.406 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1698: pg_scrub: get_last_scrub_stamp 1.d 2019-12-21T15:47:36.406 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.d 2019-12-21T15:47:36.406 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:36.407 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:36.407 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.d") | .last_scrub_stamp' 2019-12-21T15:47:36.407 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1698: pg_scrub: local last_scrub=2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:36.407 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1699: pg_scrub: ceph pg scrub 1.d 2019-12-21T15:47:36.407 INFO:tasks.workunit.client.0.smithi172.stderr:25813: instructing pg 1.d on osd.7 to scrub 2019-12-21T15:47:36.407 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1700: pg_scrub: wait_for_scrub 1.d 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:36.407 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1789: wait_for_scrub: local pgid=1.d 2019-12-21T15:47:36.408 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1790: wait_for_scrub: local last_scrub=2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:36.408 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1791: wait_for_scrub: local sname=last_scrub_stamp 2019-12-21T15:47:36.408 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i=0 )) 2019-12-21T15:47:36.408 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:36.408 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.d last_scrub_stamp 2019-12-21T15:47:36.408 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.d 2019-12-21T15:47:36.409 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:36.409 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:36.409 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.d") | .last_scrub_stamp' 2019-12-21T15:47:36.409 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:36.409 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:36.409 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:36.409 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:36.410 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.d last_scrub_stamp 2019-12-21T15:47:36.410 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.d 2019-12-21T15:47:36.410 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:36.410 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.d") | .last_scrub_stamp' 2019-12-21T15:47:36.410 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:36.410 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:36.410 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:36.411 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:36.414 INFO:tasks.workunit.client.0.smithi172.stderr:25813: 25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1697: pg_scrub: local pgid=1.10 2019-12-21T15:47:36.415 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1698: pg_scrub: get_last_scrub_stamp 1.10 2019-12-21T15:47:36.415 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.10 2019-12-21T15:47:36.415 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:36.415 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:36.415 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.10") | .last_scrub_stamp' 2019-12-21T15:47:36.416 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1698: pg_scrub: local last_scrub=2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:36.416 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1699: pg_scrub: ceph pg scrub 1.10 2019-12-21T15:47:36.416 INFO:tasks.workunit.client.0.smithi172.stderr:25813: instructing pg 1.10 on osd.5 to scrub 2019-12-21T15:47:36.416 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1700: pg_scrub: wait_for_scrub 1.10 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:36.416 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1789: wait_for_scrub: local pgid=1.10 2019-12-21T15:47:36.416 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1790: wait_for_scrub: local last_scrub=2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:36.417 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1791: wait_for_scrub: local sname=last_scrub_stamp 2019-12-21T15:47:36.417 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i=0 )) 2019-12-21T15:47:36.417 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:36.417 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.10 last_scrub_stamp 2019-12-21T15:47:36.417 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.10 2019-12-21T15:47:36.417 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:36.417 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:36.418 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.10") | .last_scrub_stamp' 2019-12-21T15:47:36.418 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:36.418 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:36.418 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:36.418 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:36.418 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.10 last_scrub_stamp 2019-12-21T15:47:36.418 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.10 2019-12-21T15:47:36.419 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:36.419 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.10") | .last_scrub_stamp' 2019-12-21T15:47:36.419 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:36.419 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:36.419 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:36.427 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( 25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1697: pg_scrub: local pgid=1.7 2019-12-21T15:47:36.428 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1698: pg_scrub: get_last_scrub_stamp 1.7 2019-12-21T15:47:36.428 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.7 2019-12-21T15:47:36.428 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:36.428 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.7") | .last_scrub_stamp' 2019-12-21T15:47:36.428 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:36.428 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1698: pg_scrub: local last_scrub=2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:36.429 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1699: pg_scrub: ceph pg scrub 1.7 2019-12-21T15:47:36.429 INFO:tasks.workunit.client.0.smithi172.stderr:25813: instructing pg 1.7 on osd.6 to scrub 2019-12-21T15:47:36.429 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1700: pg_scrub: wait_for_scrub 1.7 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:36.429 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1789: wait_for_scrub: local pgid=1.7 2019-12-21T15:47:36.429 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1790: wait_for_scrub: local last_scrub=2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:36.429 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1791: wait_for_scrub: local sname=last_scrub_stamp 2019-12-21T15:47:36.430 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i=0 )) 2019-12-21T15:47:36.430 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:36.430 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.7 last_scrub_stamp 2019-12-21T15:47:36.430 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.7 2019-12-21T15:47:36.430 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:36.430 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:36.430 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.7") | .last_scrub_stamp' 2019-12-21T15:47:36.431 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:36.431 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:36.431 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:36.431 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:36.431 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.7 last_scrub_stamp 2019-12-21T15:47:36.431 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.7 2019-12-21T15:47:36.431 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:36.432 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.7") | .last_scrub_stamp' 2019-12-21T15:47:36.432 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:36.432 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:36.432 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:36.432 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:36.477 INFO:tasks.workunit.client.0.smithi172.stderr:25813: 25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1697: pg_scrub: local pgid=1.19 2019-12-21T15:47:36.477 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1698: pg_scrub: get_last_scrub_stamp 1.19 2019-12-21T15:47:36.477 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.19 2019-12-21T15:47:36.477 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:36.478 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:36.478 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.19") | .last_scrub_stamp' 2019-12-21T15:47:36.478 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1698: pg_scrub: local last_scrub=2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:36.478 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1699: pg_scrub: ceph pg scrub 1.19 2019-12-21T15:47:36.478 INFO:tasks.workunit.client.0.smithi172.stderr:25813: instructing pg 1.19 on osd.3 to scrub 2019-12-21T15:47:36.479 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1700: pg_scrub: wait_for_scrub 1.19 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:36.479 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1789: wait_for_scrub: local pgid=1.19 2019-12-21T15:47:36.479 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1790: wait_for_scrub: local last_scrub=2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:36.479 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1791: wait_for_scrub: local sname=last_scrub_stamp 2019-12-21T15:47:36.481 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i=0 )) 2019-12-21T15:47:36.481 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:36.481 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.19 last_scrub_stamp 2019-12-21T15:47:36.481 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.19 2019-12-21T15:47:36.481 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:36.481 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:36.481 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.19") | .last_scrub_stamp' 2019-12-21T15:47:36.482 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:36.482 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:36.482 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:36.482 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:36.482 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.19 last_scrub_stamp 2019-12-21T15:47:36.482 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.19 2019-12-21T15:47:36.483 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:36.483 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.19") | .last_scrub_stamp' 2019-12-21T15:47:36.483 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:36.483 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:36.483 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:36.584 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( 25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1697: pg_scrub: local pgid=1.17 2019-12-21T15:47:36.584 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1698: pg_scrub: get_last_scrub_stamp 1.17 2019-12-21T15:47:36.584 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.17 2019-12-21T15:47:36.584 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:36.585 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:36.585 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.17") | .last_scrub_stamp' 2019-12-21T15:47:36.585 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1698: pg_scrub: local last_scrub=2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:36.585 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1699: pg_scrub: ceph pg scrub 1.17 2019-12-21T15:47:36.585 INFO:tasks.workunit.client.0.smithi172.stderr:25813: instructing pg 1.17 on osd.0 to scrub 2019-12-21T15:47:36.586 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1700: pg_scrub: wait_for_scrub 1.17 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:36.586 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1789: wait_for_scrub: local pgid=1.17 2019-12-21T15:47:36.586 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1790: wait_for_scrub: local last_scrub=2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:36.586 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1791: wait_for_scrub: local sname=last_scrub_stamp 2019-12-21T15:47:36.586 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i=0 )) 2019-12-21T15:47:36.586 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:36.586 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.17 last_scrub_stamp 2019-12-21T15:47:36.587 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.17 2019-12-21T15:47:36.587 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:36.587 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:36.587 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.17") | .last_scrub_stamp' 2019-12-21T15:47:36.587 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:36.587 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:36.588 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:36.588 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:36.588 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.17 last_scrub_stamp 2019-12-21T15:47:36.588 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.17 2019-12-21T15:47:36.588 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:36.588 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.17") | .last_scrub_stamp' 2019-12-21T15:47:36.589 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:36.589 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:36.589 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:36.632 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( 25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1697: pg_scrub: local pgid=1.c 2019-12-21T15:47:36.632 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1698: pg_scrub: get_last_scrub_stamp 1.c 2019-12-21T15:47:36.632 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.c 2019-12-21T15:47:36.632 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:36.632 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.c") | .last_scrub_stamp' 2019-12-21T15:47:36.633 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:36.633 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1698: pg_scrub: local last_scrub=2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:36.633 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1699: pg_scrub: ceph pg scrub 1.c 2019-12-21T15:47:36.633 INFO:tasks.workunit.client.0.smithi172.stderr:25813: instructing pg 1.c on osd.3 to scrub 2019-12-21T15:47:36.633 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1700: pg_scrub: wait_for_scrub 1.c 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:36.633 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1789: wait_for_scrub: local pgid=1.c 2019-12-21T15:47:36.633 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1790: wait_for_scrub: local last_scrub=2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:36.634 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1791: wait_for_scrub: local sname=last_scrub_stamp 2019-12-21T15:47:36.634 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i=0 )) 2019-12-21T15:47:36.634 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:36.634 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.c last_scrub_stamp 2019-12-21T15:47:36.634 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.c 2019-12-21T15:47:36.634 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:36.635 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:36.635 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.c") | .last_scrub_stamp' 2019-12-21T15:47:36.635 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:36.635 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:36.635 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:36.635 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:36.636 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.c last_scrub_stamp 2019-12-21T15:47:36.636 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.c 2019-12-21T15:47:36.636 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:36.636 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:36.636 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.c") | .last_scrub_stamp' 2019-12-21T15:47:36.636 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:36.637 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:36.637 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:36.657 INFO:tasks.workunit.client.0.smithi172.stderr:25813: 25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1697: pg_scrub: local pgid=1.12 2019-12-21T15:47:36.657 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1698: pg_scrub: get_last_scrub_stamp 1.12 2019-12-21T15:47:36.657 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.12 2019-12-21T15:47:36.658 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:36.658 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.12") | .last_scrub_stamp' 2019-12-21T15:47:36.658 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:36.658 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1698: pg_scrub: local last_scrub=2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:36.658 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1699: pg_scrub: ceph pg scrub 1.12 2019-12-21T15:47:36.658 INFO:tasks.workunit.client.0.smithi172.stderr:25813: instructing pg 1.12 on osd.0 to scrub 2019-12-21T15:47:36.659 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1700: pg_scrub: wait_for_scrub 1.12 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:36.659 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1789: wait_for_scrub: local pgid=1.12 2019-12-21T15:47:36.659 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1790: wait_for_scrub: local last_scrub=2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:36.659 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1791: wait_for_scrub: local sname=last_scrub_stamp 2019-12-21T15:47:36.659 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i=0 )) 2019-12-21T15:47:36.659 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:36.659 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.12 last_scrub_stamp 2019-12-21T15:47:36.660 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.12 2019-12-21T15:47:36.660 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:36.660 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:36.660 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.12") | .last_scrub_stamp' 2019-12-21T15:47:36.661 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:36.661 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:36.661 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:36.661 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:36.661 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.12 last_scrub_stamp 2019-12-21T15:47:36.661 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.12 2019-12-21T15:47:36.661 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:36.662 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:36.662 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.12") | .last_scrub_stamp' 2019-12-21T15:47:36.662 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:36.662 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:36.666 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( 25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1697: pg_scrub: local pgid=1.a 2019-12-21T15:47:36.666 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1698: pg_scrub: get_last_scrub_stamp 1.a 2019-12-21T15:47:36.667 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.a 2019-12-21T15:47:36.667 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:36.667 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.a") | .last_scrub_stamp' 2019-12-21T15:47:36.667 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:36.667 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1698: pg_scrub: local last_scrub=2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:36.667 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1699: pg_scrub: ceph pg scrub 1.a 2019-12-21T15:47:36.668 INFO:tasks.workunit.client.0.smithi172.stderr:25813: instructing pg 1.a on osd.1 to scrub 2019-12-21T15:47:36.668 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1700: pg_scrub: wait_for_scrub 1.a 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:36.668 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1789: wait_for_scrub: local pgid=1.a 2019-12-21T15:47:36.668 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1790: wait_for_scrub: local last_scrub=2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:36.668 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1791: wait_for_scrub: local sname=last_scrub_stamp 2019-12-21T15:47:36.669 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i=0 )) 2019-12-21T15:47:36.669 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:36.669 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.a last_scrub_stamp 2019-12-21T15:47:36.669 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.a 2019-12-21T15:47:36.669 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:36.669 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:36.669 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.a") | .last_scrub_stamp' 2019-12-21T15:47:36.670 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:36.670 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:36.670 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:36.670 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:36.670 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.a last_scrub_stamp 2019-12-21T15:47:36.670 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.a 2019-12-21T15:47:36.670 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:36.671 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.a") | .last_scrub_stamp' 2019-12-21T15:47:36.671 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:36.671 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:36.671 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:36.671 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:36.682 INFO:tasks.workunit.client.0.smithi172.stderr:25813: 25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1697: pg_scrub: local pgid=1.1d 2019-12-21T15:47:36.682 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1698: pg_scrub: get_last_scrub_stamp 1.1d 2019-12-21T15:47:36.682 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.1d 2019-12-21T15:47:36.682 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:36.682 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1d") | .last_scrub_stamp' 2019-12-21T15:47:36.683 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:36.683 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1698: pg_scrub: local last_scrub=2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:36.683 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1699: pg_scrub: ceph pg scrub 1.1d 2019-12-21T15:47:36.683 INFO:tasks.workunit.client.0.smithi172.stderr:25813: instructing pg 1.1d on osd.1 to scrub 2019-12-21T15:47:36.683 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1700: pg_scrub: wait_for_scrub 1.1d 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:36.683 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1789: wait_for_scrub: local pgid=1.1d 2019-12-21T15:47:36.683 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1790: wait_for_scrub: local last_scrub=2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:36.684 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1791: wait_for_scrub: local sname=last_scrub_stamp 2019-12-21T15:47:36.684 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i=0 )) 2019-12-21T15:47:36.684 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:36.684 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.1d last_scrub_stamp 2019-12-21T15:47:36.684 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.1d 2019-12-21T15:47:36.684 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:36.684 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:36.685 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1d") | .last_scrub_stamp' 2019-12-21T15:47:36.685 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:36.685 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:36.685 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:36.685 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:36.685 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.1d last_scrub_stamp 2019-12-21T15:47:36.685 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.1d 2019-12-21T15:47:36.686 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:36.686 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1d") | .last_scrub_stamp' 2019-12-21T15:47:36.686 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:36.686 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:36.686 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:36.781 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( 25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1697: pg_scrub: local pgid=1.18 2019-12-21T15:47:36.781 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1698: pg_scrub: get_last_scrub_stamp 1.18 2019-12-21T15:47:36.782 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.18 2019-12-21T15:47:36.782 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:36.782 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.18") | .last_scrub_stamp' 2019-12-21T15:47:36.782 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:36.782 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1698: pg_scrub: local last_scrub=2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:36.782 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1699: pg_scrub: ceph pg scrub 1.18 2019-12-21T15:47:36.783 INFO:tasks.workunit.client.0.smithi172.stderr:25813: instructing pg 1.18 on osd.5 to scrub 2019-12-21T15:47:36.783 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1700: pg_scrub: wait_for_scrub 1.18 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:36.783 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1789: wait_for_scrub: local pgid=1.18 2019-12-21T15:47:36.783 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1790: wait_for_scrub: local last_scrub=2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:36.783 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1791: wait_for_scrub: local sname=last_scrub_stamp 2019-12-21T15:47:36.783 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i=0 )) 2019-12-21T15:47:36.783 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:36.784 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.18 last_scrub_stamp 2019-12-21T15:47:36.784 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.18 2019-12-21T15:47:36.784 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:36.784 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:36.784 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.18") | .last_scrub_stamp' 2019-12-21T15:47:36.784 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:36.784 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:36.785 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:36.785 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:36.785 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.18 last_scrub_stamp 2019-12-21T15:47:36.785 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.18 2019-12-21T15:47:36.785 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:36.785 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:36.785 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.18") | .last_scrub_stamp' 2019-12-21T15:47:36.786 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:36.786 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:36.820 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( 25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1697: pg_scrub: local pgid=1.11 2019-12-21T15:47:36.821 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1698: pg_scrub: get_last_scrub_stamp 1.11 2019-12-21T15:47:36.821 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.11 2019-12-21T15:47:36.821 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:36.821 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:36.821 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.11") | .last_scrub_stamp' 2019-12-21T15:47:36.821 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1698: pg_scrub: local last_scrub=2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:36.822 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1699: pg_scrub: ceph pg scrub 1.11 2019-12-21T15:47:36.822 INFO:tasks.workunit.client.0.smithi172.stderr:25813: instructing pg 1.11 on osd.0 to scrub 2019-12-21T15:47:36.822 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1700: pg_scrub: wait_for_scrub 1.11 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:36.822 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1789: wait_for_scrub: local pgid=1.11 2019-12-21T15:47:36.822 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1790: wait_for_scrub: local last_scrub=2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:36.822 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1791: wait_for_scrub: local sname=last_scrub_stamp 2019-12-21T15:47:36.822 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i=0 )) 2019-12-21T15:47:36.823 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:36.823 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.11 last_scrub_stamp 2019-12-21T15:47:36.823 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.11 2019-12-21T15:47:36.823 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:36.823 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:36.823 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.11") | .last_scrub_stamp' 2019-12-21T15:47:36.824 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:36.824 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:36.824 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:36.824 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:36.824 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.11 last_scrub_stamp 2019-12-21T15:47:36.824 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.11 2019-12-21T15:47:36.824 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:36.825 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.11") | .last_scrub_stamp' 2019-12-21T15:47:36.825 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:36.825 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:36.825 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:36.825 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( 25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1697: pg_scrub: local pgid=1.1e 2019-12-21T15:47:36.826 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1698: pg_scrub: get_last_scrub_stamp 1.1e 2019-12-21T15:47:36.826 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.1e 2019-12-21T15:47:36.826 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:36.826 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1e") | .last_scrub_stamp' 2019-12-21T15:47:36.826 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:36.826 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1698: pg_scrub: local last_scrub=2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:36.826 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1699: pg_scrub: ceph pg scrub 1.1e 2019-12-21T15:47:36.827 INFO:tasks.workunit.client.0.smithi172.stderr:25813: instructing pg 1.1e on osd.3 to scrub 2019-12-21T15:47:36.827 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1700: pg_scrub: wait_for_scrub 1.1e 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:36.827 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1789: wait_for_scrub: local pgid=1.1e 2019-12-21T15:47:36.827 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1790: wait_for_scrub: local last_scrub=2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:36.827 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1791: wait_for_scrub: local sname=last_scrub_stamp 2019-12-21T15:47:36.827 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i=0 )) 2019-12-21T15:47:36.827 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:36.828 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.1e last_scrub_stamp 2019-12-21T15:47:36.828 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.1e 2019-12-21T15:47:36.828 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:36.828 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:36.828 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1e") | .last_scrub_stamp' 2019-12-21T15:47:36.828 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:36.828 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:36.829 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:36.829 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:36.829 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.1e last_scrub_stamp 2019-12-21T15:47:36.829 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.1e 2019-12-21T15:47:36.829 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:36.829 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:36.829 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1e") | .last_scrub_stamp' 2019-12-21T15:47:36.830 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:36.830 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:36.839 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( 25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1697: pg_scrub: local pgid=1.1c 2019-12-21T15:47:36.839 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1698: pg_scrub: get_last_scrub_stamp 1.1c 2019-12-21T15:47:36.840 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.1c 2019-12-21T15:47:36.840 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:36.840 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:36.840 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1c") | .last_scrub_stamp' 2019-12-21T15:47:36.840 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1698: pg_scrub: local last_scrub=2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:36.840 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1699: pg_scrub: ceph pg scrub 1.1c 2019-12-21T15:47:36.841 INFO:tasks.workunit.client.0.smithi172.stderr:25813: instructing pg 1.1c on osd.0 to scrub 2019-12-21T15:47:36.841 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1700: pg_scrub: wait_for_scrub 1.1c 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:36.841 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1789: wait_for_scrub: local pgid=1.1c 2019-12-21T15:47:36.841 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1790: wait_for_scrub: local last_scrub=2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:36.841 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1791: wait_for_scrub: local sname=last_scrub_stamp 2019-12-21T15:47:36.841 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i=0 )) 2019-12-21T15:47:36.841 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:36.842 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.1c last_scrub_stamp 2019-12-21T15:47:36.842 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.1c 2019-12-21T15:47:36.842 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:36.842 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1c") | .last_scrub_stamp' 2019-12-21T15:47:36.842 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:36.842 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:36.842 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:36.843 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:36.843 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:36.843 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.1c last_scrub_stamp 2019-12-21T15:47:36.843 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.1c 2019-12-21T15:47:36.843 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:36.843 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:36.843 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1c") | .last_scrub_stamp' 2019-12-21T15:47:36.844 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:36.844 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:36.876 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( 25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1697: pg_scrub: local pgid=1.14 2019-12-21T15:47:36.877 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1698: pg_scrub: get_last_scrub_stamp 1.14 2019-12-21T15:47:36.877 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.14 2019-12-21T15:47:36.877 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:36.877 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.14") | .last_scrub_stamp' 2019-12-21T15:47:36.877 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:36.878 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1698: pg_scrub: local last_scrub=2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:36.878 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1699: pg_scrub: ceph pg scrub 1.14 2019-12-21T15:47:36.878 INFO:tasks.workunit.client.0.smithi172.stderr:25813: instructing pg 1.14 on osd.2 to scrub 2019-12-21T15:47:36.878 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1700: pg_scrub: wait_for_scrub 1.14 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:36.878 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1789: wait_for_scrub: local pgid=1.14 2019-12-21T15:47:36.879 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1790: wait_for_scrub: local last_scrub=2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:36.879 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1791: wait_for_scrub: local sname=last_scrub_stamp 2019-12-21T15:47:36.879 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i=0 )) 2019-12-21T15:47:36.879 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:36.879 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.14 last_scrub_stamp 2019-12-21T15:47:36.879 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.14 2019-12-21T15:47:36.880 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:36.880 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:36.880 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.14") | .last_scrub_stamp' 2019-12-21T15:47:36.880 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:36.880 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:36.881 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:36.881 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:36.881 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.14 last_scrub_stamp 2019-12-21T15:47:36.881 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.14 2019-12-21T15:47:36.881 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:36.881 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:36.881 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.14") | .last_scrub_stamp' 2019-12-21T15:47:36.882 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:36.882 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:36.890 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( 25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1697: pg_scrub: local pgid=1.1a 2019-12-21T15:47:36.890 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1698: pg_scrub: get_last_scrub_stamp 1.1a 2019-12-21T15:47:36.891 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.1a 2019-12-21T15:47:36.891 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:36.891 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1a") | .last_scrub_stamp' 2019-12-21T15:47:36.891 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:36.891 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1698: pg_scrub: local last_scrub=2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:36.891 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1699: pg_scrub: ceph pg scrub 1.1a 2019-12-21T15:47:36.891 INFO:tasks.workunit.client.0.smithi172.stderr:25813: instructing pg 1.1a on osd.1 to scrub 2019-12-21T15:47:36.892 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1700: pg_scrub: wait_for_scrub 1.1a 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:36.892 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1789: wait_for_scrub: local pgid=1.1a 2019-12-21T15:47:36.892 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1790: wait_for_scrub: local last_scrub=2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:36.892 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1791: wait_for_scrub: local sname=last_scrub_stamp 2019-12-21T15:47:36.892 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i=0 )) 2019-12-21T15:47:36.892 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:36.893 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.1a last_scrub_stamp 2019-12-21T15:47:36.893 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.1a 2019-12-21T15:47:36.893 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:36.893 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:36.893 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1a") | .last_scrub_stamp' 2019-12-21T15:47:36.893 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:36.893 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:36.894 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:36.894 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:36.894 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.1a last_scrub_stamp 2019-12-21T15:47:36.894 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.1a 2019-12-21T15:47:36.894 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:36.894 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1a") | .last_scrub_stamp' 2019-12-21T15:47:36.894 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:36.895 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:36.895 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:36.895 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( 25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1697: pg_scrub: local pgid=1.1b 2019-12-21T15:47:36.895 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1698: pg_scrub: get_last_scrub_stamp 1.1b 2019-12-21T15:47:36.896 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.1b 2019-12-21T15:47:36.896 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:36.896 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:36.896 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1b") | .last_scrub_stamp' 2019-12-21T15:47:36.896 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1698: pg_scrub: local last_scrub=2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:36.896 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1699: pg_scrub: ceph pg scrub 1.1b 2019-12-21T15:47:36.896 INFO:tasks.workunit.client.0.smithi172.stderr:25813: instructing pg 1.1b on osd.0 to scrub 2019-12-21T15:47:36.897 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1700: pg_scrub: wait_for_scrub 1.1b 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:36.897 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1789: wait_for_scrub: local pgid=1.1b 2019-12-21T15:47:36.897 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1790: wait_for_scrub: local last_scrub=2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:36.897 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1791: wait_for_scrub: local sname=last_scrub_stamp 2019-12-21T15:47:36.897 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i=0 )) 2019-12-21T15:47:36.897 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:36.898 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.1b last_scrub_stamp 2019-12-21T15:47:36.898 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.1b 2019-12-21T15:47:36.898 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:36.898 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:36.898 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1b") | .last_scrub_stamp' 2019-12-21T15:47:36.898 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:36.898 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:36.899 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:36.899 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:36.899 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.1b last_scrub_stamp 2019-12-21T15:47:36.899 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.1b 2019-12-21T15:47:36.899 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:36.899 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1b") | .last_scrub_stamp' 2019-12-21T15:47:36.899 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:36.900 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:36.900 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:36.904 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( 25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1697: pg_scrub: local pgid=1.1f 2019-12-21T15:47:36.905 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1698: pg_scrub: get_last_scrub_stamp 1.1f 2019-12-21T15:47:36.905 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.1f 2019-12-21T15:47:36.905 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:36.905 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1f") | .last_scrub_stamp' 2019-12-21T15:47:36.905 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:36.905 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1698: pg_scrub: local last_scrub=2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:36.906 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1699: pg_scrub: ceph pg scrub 1.1f 2019-12-21T15:47:36.906 INFO:tasks.workunit.client.0.smithi172.stderr:25813: instructing pg 1.1f on osd.5 to scrub 2019-12-21T15:47:36.906 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1700: pg_scrub: wait_for_scrub 1.1f 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:36.906 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1789: wait_for_scrub: local pgid=1.1f 2019-12-21T15:47:36.906 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1790: wait_for_scrub: local last_scrub=2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:36.906 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1791: wait_for_scrub: local sname=last_scrub_stamp 2019-12-21T15:47:36.906 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i=0 )) 2019-12-21T15:47:36.907 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:36.907 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.1f last_scrub_stamp 2019-12-21T15:47:36.907 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.1f 2019-12-21T15:47:36.907 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:36.907 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1f") | .last_scrub_stamp' 2019-12-21T15:47:36.907 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:36.907 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:36.908 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:36.908 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:36.908 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:36.908 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.1f last_scrub_stamp 2019-12-21T15:47:36.908 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.1f 2019-12-21T15:47:36.908 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:36.909 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1f") | .last_scrub_stamp' 2019-12-21T15:47:36.909 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:36.909 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:36.909 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:36.928 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( 25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1697: pg_scrub: local pgid=1.15 2019-12-21T15:47:36.928 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1698: pg_scrub: get_last_scrub_stamp 1.15 2019-12-21T15:47:36.928 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.15 2019-12-21T15:47:36.929 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:36.929 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:36.929 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.15") | .last_scrub_stamp' 2019-12-21T15:47:36.929 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1698: pg_scrub: local last_scrub=2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:36.929 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1699: pg_scrub: ceph pg scrub 1.15 2019-12-21T15:47:36.929 INFO:tasks.workunit.client.0.smithi172.stderr:25813: instructing pg 1.15 on osd.1 to scrub 2019-12-21T15:47:36.929 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1700: pg_scrub: wait_for_scrub 1.15 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:36.930 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1789: wait_for_scrub: local pgid=1.15 2019-12-21T15:47:36.930 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1790: wait_for_scrub: local last_scrub=2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:36.930 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1791: wait_for_scrub: local sname=last_scrub_stamp 2019-12-21T15:47:36.930 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i=0 )) 2019-12-21T15:47:36.930 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:36.930 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.15 last_scrub_stamp 2019-12-21T15:47:36.930 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.15 2019-12-21T15:47:36.931 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:36.931 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.15") | .last_scrub_stamp' 2019-12-21T15:47:36.931 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:36.931 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:36.931 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:36.931 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:36.931 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:36.932 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.15 last_scrub_stamp 2019-12-21T15:47:36.932 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.15 2019-12-21T15:47:36.932 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:36.932 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.15") | .last_scrub_stamp' 2019-12-21T15:47:36.932 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:36.932 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:36.932 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:36.936 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( 25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1697: pg_scrub: local pgid=1.16 2019-12-21T15:47:36.936 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1698: pg_scrub: get_last_scrub_stamp 1.16 2019-12-21T15:47:36.936 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.16 2019-12-21T15:47:36.937 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:36.937 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:36.937 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.16") | .last_scrub_stamp' 2019-12-21T15:47:36.937 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1698: pg_scrub: local last_scrub=2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:36.937 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1699: pg_scrub: ceph pg scrub 1.16 2019-12-21T15:47:36.937 INFO:tasks.workunit.client.0.smithi172.stderr:25813: instructing pg 1.16 on osd.6 to scrub 2019-12-21T15:47:36.937 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1700: pg_scrub: wait_for_scrub 1.16 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:36.938 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1789: wait_for_scrub: local pgid=1.16 2019-12-21T15:47:36.938 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1790: wait_for_scrub: local last_scrub=2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:36.938 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1791: wait_for_scrub: local sname=last_scrub_stamp 2019-12-21T15:47:36.938 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i=0 )) 2019-12-21T15:47:36.938 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:36.938 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.16 last_scrub_stamp 2019-12-21T15:47:36.938 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.16 2019-12-21T15:47:36.939 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:36.939 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.16") | .last_scrub_stamp' 2019-12-21T15:47:36.939 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:36.939 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:36.939 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:36.939 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:36.940 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:36.940 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.16 last_scrub_stamp 2019-12-21T15:47:36.940 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.16 2019-12-21T15:47:36.940 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:36.940 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.16") | .last_scrub_stamp' 2019-12-21T15:47:36.940 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:36.940 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:36.941 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:38.602 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:38.602 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2019-12-21T15:47:38.603 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.3 2019-12-21T15:47:38.603 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:38.603 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:38.603 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2019-12-21T15:47:38.603 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:38.603 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:38.604 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:38.604 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:38.604 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2019-12-21T15:47:38.604 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.3 2019-12-21T15:47:38.604 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:38.604 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:38.605 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2019-12-21T15:47:38.605 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:35.432862+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:38.605 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1795: wait_for_scrub: return 0 2019-12-21T15:47:38.605 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1911: run_in_background: return 0 2019-12-21T15:47:40.867 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:40.867 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.6 last_scrub_stamp 2019-12-21T15:47:40.867 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.6 2019-12-21T15:47:40.867 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:40.868 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:40.868 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.6") | .last_scrub_stamp' 2019-12-21T15:47:40.868 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:40.868 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:40.868 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:40.868 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:40.868 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.6 last_scrub_stamp 2019-12-21T15:47:40.869 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.6 2019-12-21T15:47:40.869 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:40.869 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:40.869 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.6") | .last_scrub_stamp' 2019-12-21T15:47:40.869 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:40.869 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:40.870 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:40.870 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:40.870 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.6 last_scrub_stamp 2019-12-21T15:47:40.870 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.6 2019-12-21T15:47:40.870 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:40.870 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:40.870 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.6") | .last_scrub_stamp' 2019-12-21T15:47:40.871 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:37.080383+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:40.871 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1795: wait_for_scrub: return 0 2019-12-21T15:47:40.871 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1911: run_in_background: return 0 2019-12-21T15:47:41.259 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:41.259 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.2 last_scrub_stamp 2019-12-21T15:47:41.259 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.2 2019-12-21T15:47:41.259 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:41.259 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:41.259 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.2") | .last_scrub_stamp' 2019-12-21T15:47:41.260 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:41.260 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:41.260 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:41.260 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:41.260 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.2 last_scrub_stamp 2019-12-21T15:47:41.260 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.2 2019-12-21T15:47:41.260 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:41.261 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:41.261 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.2") | .last_scrub_stamp' 2019-12-21T15:47:41.261 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:41.261 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:41.261 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:41.261 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:41.261 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.2 last_scrub_stamp 2019-12-21T15:47:41.262 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.2 2019-12-21T15:47:41.262 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:41.262 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:41.262 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.2") | .last_scrub_stamp' 2019-12-21T15:47:41.262 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:41.262 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:41.263 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:41.263 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:41.263 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.2 last_scrub_stamp 2019-12-21T15:47:41.263 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.2 2019-12-21T15:47:41.263 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:41.357 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephte/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:41.357 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.9 last_scrub_stamp 2019-12-21T15:47:41.357 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.9 2019-12-21T15:47:41.357 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:41.357 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:41.357 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.9") | .last_scrub_stamp' 2019-12-21T15:47:41.358 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:41.358 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:41.358 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:41.358 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:41.358 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.9 last_scrub_stamp 2019-12-21T15:47:41.358 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.9 2019-12-21T15:47:41.358 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:41.359 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.9") | .last_scrub_stamp' 2019-12-21T15:47:41.359 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:41.359 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:41.359 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:41.359 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:41.359 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:41.359 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.9 last_scrub_stamp 2019-12-21T15:47:41.360 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.9 2019-12-21T15:47:41.360 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:41.360 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.9") | .last_scrub_stamp' 2019-12-21T15:47:41.360 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:41.360 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:41.360 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:41.361 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:41.361 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:41.361 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.9 last_scrub_stamp 2019-12-21T15:47:41.361 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.9 2019-12-21T15:47:41.361 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:41.368 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephte/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:41.368 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2019-12-21T15:47:41.368 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.0 2019-12-21T15:47:41.368 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:41.368 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:41.369 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2019-12-21T15:47:41.369 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:41.369 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:41.369 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:41.369 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:41.369 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2019-12-21T15:47:41.369 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.0 2019-12-21T15:47:41.370 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:41.370 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2019-12-21T15:47:41.370 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:41.370 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:41.370 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:41.370 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:41.370 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:41.371 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2019-12-21T15:47:41.371 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.0 2019-12-21T15:47:41.371 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:41.371 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:41.371 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2019-12-21T15:47:41.371 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:41.371 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:41.372 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:41.372 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:41.372 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2019-12-21T15:47:41.372 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.0 2019-12-21T15:47:41.372 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:41.427 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephte/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:41.427 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.5 last_scrub_stamp 2019-12-21T15:47:41.428 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.5 2019-12-21T15:47:41.428 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:41.428 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:41.428 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.5") | .last_scrub_stamp' 2019-12-21T15:47:41.428 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:41.429 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:41.429 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:41.429 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:41.429 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.5 last_scrub_stamp 2019-12-21T15:47:41.429 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.5 2019-12-21T15:47:41.429 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:41.429 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:41.429 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.5") | .last_scrub_stamp' 2019-12-21T15:47:41.430 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:41.430 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:41.430 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:41.430 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:41.430 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.5 last_scrub_stamp 2019-12-21T15:47:41.430 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.5 2019-12-21T15:47:41.431 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:41.431 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:41.431 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.5") | .last_scrub_stamp' 2019-12-21T15:47:41.431 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:41.431 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:41.431 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:41.431 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:41.432 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.5 last_scrub_stamp 2019-12-21T15:47:41.432 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.5 2019-12-21T15:47:41.432 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:41.556 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephte/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:41.556 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.e last_scrub_stamp 2019-12-21T15:47:41.556 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.e 2019-12-21T15:47:41.556 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:41.556 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:41.557 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.e") | .last_scrub_stamp' 2019-12-21T15:47:41.557 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:41.557 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:41.557 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:41.557 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:41.557 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.e last_scrub_stamp 2019-12-21T15:47:41.558 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.e 2019-12-21T15:47:41.558 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:41.558 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:41.558 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.e") | .last_scrub_stamp' 2019-12-21T15:47:41.558 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:41.558 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:41.558 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:41.559 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:41.559 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.e last_scrub_stamp 2019-12-21T15:47:41.559 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.e 2019-12-21T15:47:41.559 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:41.559 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:41.559 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.e") | .last_scrub_stamp' 2019-12-21T15:47:41.559 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:41.560 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:41.560 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:41.560 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:41.560 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.e last_scrub_stamp 2019-12-21T15:47:41.560 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.e 2019-12-21T15:47:41.560 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:41.778 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtei++ )) 2019-12-21T15:47:41.778 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:41.778 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.13 last_scrub_stamp 2019-12-21T15:47:41.778 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.13 2019-12-21T15:47:41.779 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:41.779 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:41.779 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.13") | .last_scrub_stamp' 2019-12-21T15:47:41.779 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:41.779 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:41.779 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:41.780 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:41.780 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.13 last_scrub_stamp 2019-12-21T15:47:41.780 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.13 2019-12-21T15:47:41.780 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:41.780 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:41.780 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.13") | .last_scrub_stamp' 2019-12-21T15:47:41.780 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:41.781 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:41.781 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:41.781 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:41.781 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.13 last_scrub_stamp 2019-12-21T15:47:41.781 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.13 2019-12-21T15:47:41.781 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:41.781 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.13") | .last_scrub_stamp' 2019-12-21T15:47:41.782 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:41.782 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:41.782 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:41.782 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:41.782 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:41.782 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.13 last_scrub_stamp 2019-12-21T15:47:41.783 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.13 2019-12-21T15:47:41.783 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:41.821 INFO:tasks.workunit.client.0.smithi172.stderr:25i++ )) 2019-12-21T15:47:41.821 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:41.821 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.10 last_scrub_stamp 2019-12-21T15:47:41.821 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.10 2019-12-21T15:47:41.822 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:41.822 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:41.822 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.10") | .last_scrub_stamp' 2019-12-21T15:47:41.822 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:41.822 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:41.822 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:41.822 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:41.823 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.10 last_scrub_stamp 2019-12-21T15:47:41.823 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.10 2019-12-21T15:47:41.823 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:41.823 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.10") | .last_scrub_stamp' 2019-12-21T15:47:41.823 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:41.823 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:41.823 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:41.824 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:41.824 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:41.824 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.10 last_scrub_stamp 2019-12-21T15:47:41.824 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.10 2019-12-21T15:47:41.824 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:41.824 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:41.824 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.10") | .last_scrub_stamp' 2019-12-21T15:47:41.825 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:41.825 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:41.825 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:41.825 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:41.825 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.10 last_scrub_stamp 2019-12-21T15:47:41.825 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.10 2019-12-21T15:47:41.825 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:41.833 INFO:tasks.workunit.client.0.smithi172.stderr:25/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:41.833 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.4 last_scrub_stamp 2019-12-21T15:47:41.833 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.4 2019-12-21T15:47:41.833 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:41.834 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:41.834 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.4") | .last_scrub_stamp' 2019-12-21T15:47:41.834 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:41.834 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:41.834 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:41.834 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:41.834 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.4 last_scrub_stamp 2019-12-21T15:47:41.835 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.4 2019-12-21T15:47:41.835 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:41.835 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:41.835 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.4") | .last_scrub_stamp' 2019-12-21T15:47:41.835 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:41.835 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:41.835 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:41.836 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:41.836 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.4 last_scrub_stamp 2019-12-21T15:47:41.836 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.4 2019-12-21T15:47:41.836 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:41.837 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.4") | .last_scrub_stamp' 2019-12-21T15:47:41.837 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:41.837 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:41.837 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:41.837 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:41.838 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:41.838 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.4 last_scrub_stamp 2019-12-21T15:47:41.838 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.4 2019-12-21T15:47:41.838 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:41.857 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephte/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:41.858 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.1 last_scrub_stamp 2019-12-21T15:47:41.858 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.1 2019-12-21T15:47:41.858 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:41.858 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:41.858 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1") | .last_scrub_stamp' 2019-12-21T15:47:41.858 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:41.859 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:41.859 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:41.859 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:41.859 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.1 last_scrub_stamp 2019-12-21T15:47:41.859 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.1 2019-12-21T15:47:41.859 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:41.859 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:41.860 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1") | .last_scrub_stamp' 2019-12-21T15:47:41.860 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:41.860 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:41.860 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:41.860 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:41.860 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.1 last_scrub_stamp 2019-12-21T15:47:41.860 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.1 2019-12-21T15:47:41.861 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:41.861 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:41.861 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1") | .last_scrub_stamp' 2019-12-21T15:47:41.861 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:41.861 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:41.861 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:41.861 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:41.862 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.1 last_scrub_stamp 2019-12-21T15:47:41.862 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.1 2019-12-21T15:47:41.862 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:41.866 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephte/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:41.866 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.c last_scrub_stamp 2019-12-21T15:47:41.866 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.c 2019-12-21T15:47:41.866 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:41.867 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.c") | .last_scrub_stamp' 2019-12-21T15:47:41.867 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:41.867 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:41.867 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:41.867 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:41.867 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:41.867 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.c last_scrub_stamp 2019-12-21T15:47:41.868 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.c 2019-12-21T15:47:41.868 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:41.868 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:41.868 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.c") | .last_scrub_stamp' 2019-12-21T15:47:41.868 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:41.868 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:41.868 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:41.869 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:41.869 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.c last_scrub_stamp 2019-12-21T15:47:41.869 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.c 2019-12-21T15:47:41.869 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:41.869 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:41.869 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.c") | .last_scrub_stamp' 2019-12-21T15:47:41.869 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:41.870 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:41.870 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:41.870 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:41.870 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.c last_scrub_stamp 2019-12-21T15:47:41.870 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.c 2019-12-21T15:47:41.870 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:41.904 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephte/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:41.905 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.b last_scrub_stamp 2019-12-21T15:47:41.905 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.b 2019-12-21T15:47:41.905 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:41.905 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:41.905 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.b") | .last_scrub_stamp' 2019-12-21T15:47:41.905 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:41.905 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:41.906 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:41.906 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:41.906 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.b last_scrub_stamp 2019-12-21T15:47:41.906 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.b 2019-12-21T15:47:41.906 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:41.906 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.b") | .last_scrub_stamp' 2019-12-21T15:47:41.906 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:41.907 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:41.907 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:41.907 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:41.907 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:41.907 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.b last_scrub_stamp 2019-12-21T15:47:41.907 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.b 2019-12-21T15:47:41.907 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:41.908 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.b") | .last_scrub_stamp' 2019-12-21T15:47:41.908 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:41.908 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:41.908 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:41.908 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:41.908 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:41.908 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.b last_scrub_stamp 2019-12-21T15:47:41.909 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.b 2019-12-21T15:47:41.909 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:41.909 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephte/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:41.909 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.8 last_scrub_stamp 2019-12-21T15:47:41.910 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.8 2019-12-21T15:47:41.910 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:41.910 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:41.910 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.8") | .last_scrub_stamp' 2019-12-21T15:47:41.910 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:41.910 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:41.910 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:41.911 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:41.911 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.8 last_scrub_stamp 2019-12-21T15:47:41.911 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.8 2019-12-21T15:47:41.911 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:41.911 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:41.911 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.8") | .last_scrub_stamp' 2019-12-21T15:47:41.911 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:41.911 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:41.912 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:41.912 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:41.912 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.8 last_scrub_stamp 2019-12-21T15:47:41.912 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.8 2019-12-21T15:47:41.912 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:41.912 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:41.913 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.8") | .last_scrub_stamp' 2019-12-21T15:47:41.913 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:41.913 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:41.913 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:41.913 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:41.913 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.8 last_scrub_stamp 2019-12-21T15:47:41.913 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.8 2019-12-21T15:47:41.914 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:41.951 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephte/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:41.951 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.7 last_scrub_stamp 2019-12-21T15:47:41.951 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.7 2019-12-21T15:47:41.951 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:41.951 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.7") | .last_scrub_stamp' 2019-12-21T15:47:41.952 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:41.952 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:41.952 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:41.952 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:41.952 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:41.952 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.7 last_scrub_stamp 2019-12-21T15:47:41.952 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.7 2019-12-21T15:47:41.953 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:41.953 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:41.953 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.7") | .last_scrub_stamp' 2019-12-21T15:47:41.953 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:41.953 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:41.953 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:41.953 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:41.954 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.7 last_scrub_stamp 2019-12-21T15:47:41.954 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.7 2019-12-21T15:47:41.954 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:41.954 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:41.954 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.7") | .last_scrub_stamp' 2019-12-21T15:47:41.954 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:41.954 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:41.955 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:41.955 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:41.955 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.7 last_scrub_stamp 2019-12-21T15:47:41.955 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.7 2019-12-21T15:47:41.955 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:41.967 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephte/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:41.967 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.d last_scrub_stamp 2019-12-21T15:47:41.967 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.d 2019-12-21T15:47:41.967 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:41.967 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.d") | .last_scrub_stamp' 2019-12-21T15:47:41.968 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:41.968 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:41.968 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:41.968 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:41.968 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:41.968 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.d last_scrub_stamp 2019-12-21T15:47:41.968 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.d 2019-12-21T15:47:41.969 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:41.969 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:41.969 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.d") | .last_scrub_stamp' 2019-12-21T15:47:41.969 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:41.969 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:41.969 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:41.970 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:41.970 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.d last_scrub_stamp 2019-12-21T15:47:41.970 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.d 2019-12-21T15:47:41.970 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:41.970 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:41.970 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.d") | .last_scrub_stamp' 2019-12-21T15:47:41.970 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:41.971 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:41.971 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:41.971 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:41.971 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.d last_scrub_stamp 2019-12-21T15:47:41.971 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.d 2019-12-21T15:47:41.971 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:41.981 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtei++ )) 2019-12-21T15:47:41.981 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:41.982 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.19 last_scrub_stamp 2019-12-21T15:47:41.982 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.19 2019-12-21T15:47:41.982 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:41.982 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.19") | .last_scrub_stamp' 2019-12-21T15:47:41.982 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:41.982 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:41.982 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:41.983 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:41.983 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:41.983 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.19 last_scrub_stamp 2019-12-21T15:47:41.983 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.19 2019-12-21T15:47:41.983 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:41.983 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:41.983 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.19") | .last_scrub_stamp' 2019-12-21T15:47:41.984 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:41.984 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:41.984 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:41.984 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:41.984 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.19 last_scrub_stamp 2019-12-21T15:47:41.984 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.19 2019-12-21T15:47:41.984 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:41.985 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:41.985 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.19") | .last_scrub_stamp' 2019-12-21T15:47:41.985 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:41.985 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:41.985 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:41.985 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:41.985 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.19 last_scrub_stamp 2019-12-21T15:47:41.986 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.19 2019-12-21T15:47:41.986 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:41.992 INFO:tasks.workunit.client.0.smithi172.stderr:25/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:41.992 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.f last_scrub_stamp 2019-12-21T15:47:41.992 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.f 2019-12-21T15:47:41.992 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:41.992 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.f") | .last_scrub_stamp' 2019-12-21T15:47:41.992 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:41.993 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:41.993 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:41.993 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:41.993 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:41.993 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.f last_scrub_stamp 2019-12-21T15:47:41.993 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.f 2019-12-21T15:47:41.994 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:41.994 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:41.994 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.f") | .last_scrub_stamp' 2019-12-21T15:47:41.994 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:41.994 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:41.994 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:41.994 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:41.995 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.f last_scrub_stamp 2019-12-21T15:47:41.995 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.f 2019-12-21T15:47:41.995 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:41.995 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:41.995 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.f") | .last_scrub_stamp' 2019-12-21T15:47:41.995 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:41.995 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:41.996 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:41.996 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:41.996 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.f last_scrub_stamp 2019-12-21T15:47:41.996 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.f 2019-12-21T15:47:41.996 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:42.000 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtei++ )) 2019-12-21T15:47:42.000 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:42.001 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.12 last_scrub_stamp 2019-12-21T15:47:42.001 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.12 2019-12-21T15:47:42.001 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:42.001 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.12") | .last_scrub_stamp' 2019-12-21T15:47:42.001 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:42.001 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:42.001 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:42.002 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:42.002 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:42.002 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.12 last_scrub_stamp 2019-12-21T15:47:42.002 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.12 2019-12-21T15:47:42.002 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:42.002 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.12") | .last_scrub_stamp' 2019-12-21T15:47:42.003 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:42.003 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:42.003 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:42.003 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:42.003 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:42.003 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.12 last_scrub_stamp 2019-12-21T15:47:42.003 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.12 2019-12-21T15:47:42.003 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:42.004 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:42.004 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.12") | .last_scrub_stamp' 2019-12-21T15:47:42.004 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:42.004 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:42.004 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:42.004 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:42.005 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.12 last_scrub_stamp 2019-12-21T15:47:42.005 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.12 2019-12-21T15:47:42.005 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:42.017 INFO:tasks.workunit.client.0.smithi172.stderr:25i++ )) 2019-12-21T15:47:42.017 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:42.017 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.17 last_scrub_stamp 2019-12-21T15:47:42.017 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.17 2019-12-21T15:47:42.017 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:42.017 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:42.018 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.17") | .last_scrub_stamp' 2019-12-21T15:47:42.018 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:42.018 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:42.018 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:42.018 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:42.018 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.17 last_scrub_stamp 2019-12-21T15:47:42.018 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.17 2019-12-21T15:47:42.019 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:42.019 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.17") | .last_scrub_stamp' 2019-12-21T15:47:42.019 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:42.019 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:42.019 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:42.019 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:42.019 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:42.020 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.17 last_scrub_stamp 2019-12-21T15:47:42.020 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.17 2019-12-21T15:47:42.020 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:42.020 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.17") | .last_scrub_stamp' 2019-12-21T15:47:42.020 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:42.020 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:42.020 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:42.021 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:42.021 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:42.021 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.17 last_scrub_stamp 2019-12-21T15:47:42.021 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.17 2019-12-21T15:47:42.021 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:42.026 INFO:tasks.workunit.client.0.smithi172.stderr:25i++ )) 2019-12-21T15:47:42.026 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:42.026 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.1d last_scrub_stamp 2019-12-21T15:47:42.026 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.1d 2019-12-21T15:47:42.027 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:42.027 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1d") | .last_scrub_stamp' 2019-12-21T15:47:42.027 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:42.027 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:42.027 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:42.027 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:42.028 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:42.028 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.1d last_scrub_stamp 2019-12-21T15:47:42.028 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.1d 2019-12-21T15:47:42.028 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:42.028 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1d") | .last_scrub_stamp' 2019-12-21T15:47:42.028 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:42.029 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:42.029 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:42.029 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:42.029 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:42.029 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.1d last_scrub_stamp 2019-12-21T15:47:42.029 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.1d 2019-12-21T15:47:42.029 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:42.030 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:42.030 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1d") | .last_scrub_stamp' 2019-12-21T15:47:42.030 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:42.030 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:42.030 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:42.030 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:42.030 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.1d last_scrub_stamp 2019-12-21T15:47:42.030 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.1d 2019-12-21T15:47:42.031 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:42.325 INFO:tasks.workunit.client.0.smithi172.stderr:25i++ )) 2019-12-21T15:47:42.325 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:42.325 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.1c last_scrub_stamp 2019-12-21T15:47:42.325 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.1c 2019-12-21T15:47:42.325 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:42.325 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:42.326 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1c") | .last_scrub_stamp' 2019-12-21T15:47:42.326 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:42.326 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:42.326 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:42.326 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:42.326 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.1c last_scrub_stamp 2019-12-21T15:47:42.326 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.1c 2019-12-21T15:47:42.327 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:42.327 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:42.327 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1c") | .last_scrub_stamp' 2019-12-21T15:47:42.327 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:42.327 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:42.327 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:42.327 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:42.328 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.1c last_scrub_stamp 2019-12-21T15:47:42.328 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.1c 2019-12-21T15:47:42.328 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:42.328 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1c") | .last_scrub_stamp' 2019-12-21T15:47:42.328 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:42.328 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:42.328 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:42.329 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:42.329 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:42.329 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.1c last_scrub_stamp 2019-12-21T15:47:42.329 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.1c 2019-12-21T15:47:42.329 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:42.337 INFO:tasks.workunit.client.0.smithi172.stderr:25i++ )) 2019-12-21T15:47:42.337 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:42.338 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.18 last_scrub_stamp 2019-12-21T15:47:42.338 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.18 2019-12-21T15:47:42.338 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:42.338 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.18") | .last_scrub_stamp' 2019-12-21T15:47:42.338 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:42.338 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:42.339 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:42.339 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:42.339 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:42.339 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.18 last_scrub_stamp 2019-12-21T15:47:42.339 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.18 2019-12-21T15:47:42.339 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:42.339 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:42.339 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.18") | .last_scrub_stamp' 2019-12-21T15:47:42.340 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:42.340 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:42.340 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:42.340 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:42.340 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.18 last_scrub_stamp 2019-12-21T15:47:42.340 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.18 2019-12-21T15:47:42.341 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:42.341 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:42.341 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.18") | .last_scrub_stamp' 2019-12-21T15:47:42.341 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:42.341 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:42.341 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:42.342 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:42.342 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.18 last_scrub_stamp 2019-12-21T15:47:42.342 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.18 2019-12-21T15:47:42.342 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:42.355 INFO:tasks.workunit.client.0.smithi172.stderr:25/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:42.355 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.a last_scrub_stamp 2019-12-21T15:47:42.356 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.a 2019-12-21T15:47:42.356 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:42.357 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.a") | .last_scrub_stamp' 2019-12-21T15:47:42.357 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:42.357 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:42.357 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:42.357 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:42.357 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:42.357 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.a last_scrub_stamp 2019-12-21T15:47:42.358 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.a 2019-12-21T15:47:42.358 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:42.358 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:42.358 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.a") | .last_scrub_stamp' 2019-12-21T15:47:42.358 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:42.358 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:42.358 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:42.359 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:42.359 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.a last_scrub_stamp 2019-12-21T15:47:42.359 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.a 2019-12-21T15:47:42.359 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:42.359 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:42.359 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.a") | .last_scrub_stamp' 2019-12-21T15:47:42.359 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:42.360 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:42.360 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:42.360 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:42.360 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.a last_scrub_stamp 2019-12-21T15:47:42.360 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.a 2019-12-21T15:47:42.360 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:42.393 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtei++ )) 2019-12-21T15:47:42.394 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:42.394 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.11 last_scrub_stamp 2019-12-21T15:47:42.394 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.11 2019-12-21T15:47:42.394 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:42.394 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:42.394 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.11") | .last_scrub_stamp' 2019-12-21T15:47:42.395 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:42.395 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:42.395 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:42.395 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:42.395 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.11 last_scrub_stamp 2019-12-21T15:47:42.395 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.11 2019-12-21T15:47:42.395 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:42.396 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:42.396 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.11") | .last_scrub_stamp' 2019-12-21T15:47:42.396 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:42.396 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:42.396 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:42.396 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:42.396 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.11 last_scrub_stamp 2019-12-21T15:47:42.397 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.11 2019-12-21T15:47:42.397 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:42.397 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:42.397 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.11") | .last_scrub_stamp' 2019-12-21T15:47:42.397 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:42.397 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:42.398 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:42.398 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:42.398 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.11 last_scrub_stamp 2019-12-21T15:47:42.398 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.11 2019-12-21T15:47:42.398 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:42.414 INFO:tasks.workunit.client.0.smithi172.stderr:25i++ )) 2019-12-21T15:47:42.415 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:42.415 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.1e last_scrub_stamp 2019-12-21T15:47:42.415 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.1e 2019-12-21T15:47:42.415 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:42.415 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:42.415 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1e") | .last_scrub_stamp' 2019-12-21T15:47:42.416 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:42.416 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:42.416 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:42.416 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:42.416 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.1e last_scrub_stamp 2019-12-21T15:47:42.416 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.1e 2019-12-21T15:47:42.416 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:42.417 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:42.417 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1e") | .last_scrub_stamp' 2019-12-21T15:47:42.417 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:42.417 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:42.417 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:42.417 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:42.418 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.1e last_scrub_stamp 2019-12-21T15:47:42.418 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.1e 2019-12-21T15:47:42.418 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:42.418 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1e") | .last_scrub_stamp' 2019-12-21T15:47:42.418 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:42.418 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:42.419 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:42.419 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:42.419 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:42.419 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.1e last_scrub_stamp 2019-12-21T15:47:42.419 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.1e 2019-12-21T15:47:42.419 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:42.493 INFO:tasks.workunit.client.0.smithi172.stderr:25i++ )) 2019-12-21T15:47:42.493 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:42.494 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.1f last_scrub_stamp 2019-12-21T15:47:42.494 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.1f 2019-12-21T15:47:42.494 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:42.494 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:42.495 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1f") | .last_scrub_stamp' 2019-12-21T15:47:42.495 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:42.495 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:42.495 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:42.495 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:42.495 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.1f last_scrub_stamp 2019-12-21T15:47:42.495 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.1f 2019-12-21T15:47:42.496 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:42.496 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:42.496 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1f") | .last_scrub_stamp' 2019-12-21T15:47:42.496 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:42.496 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:42.496 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:42.497 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:42.497 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.1f last_scrub_stamp 2019-12-21T15:47:42.497 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.1f 2019-12-21T15:47:42.497 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:42.497 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:42.497 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1f") | .last_scrub_stamp' 2019-12-21T15:47:42.497 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:42.498 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:42.498 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:42.498 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:42.498 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.1f last_scrub_stamp 2019-12-21T15:47:42.498 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.1f 2019-12-21T15:47:42.498 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:42.530 INFO:tasks.workunit.client.0.smithi172.stderr:25i++ )) 2019-12-21T15:47:42.530 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:42.530 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.1a last_scrub_stamp 2019-12-21T15:47:42.531 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.1a 2019-12-21T15:47:42.531 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:42.531 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:42.531 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1a") | .last_scrub_stamp' 2019-12-21T15:47:42.531 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:42.531 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:42.531 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:42.532 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:42.532 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.1a last_scrub_stamp 2019-12-21T15:47:42.532 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.1a 2019-12-21T15:47:42.532 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:42.532 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1a") | .last_scrub_stamp' 2019-12-21T15:47:42.532 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:42.532 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:42.533 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:42.533 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:42.533 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:42.533 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.1a last_scrub_stamp 2019-12-21T15:47:42.533 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.1a 2019-12-21T15:47:42.533 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:42.533 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1a") | .last_scrub_stamp' 2019-12-21T15:47:42.534 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:42.534 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:42.534 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:42.534 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:42.534 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:42.534 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.1a last_scrub_stamp 2019-12-21T15:47:42.534 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.1a 2019-12-21T15:47:42.535 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:42.546 INFO:tasks.workunit.client.0.smithi172.stderr:25i++ )) 2019-12-21T15:47:42.546 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:42.546 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.14 last_scrub_stamp 2019-12-21T15:47:42.546 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.14 2019-12-21T15:47:42.547 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:42.547 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:42.547 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.14") | .last_scrub_stamp' 2019-12-21T15:47:42.547 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:42.547 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:42.547 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:42.547 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:42.548 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.14 last_scrub_stamp 2019-12-21T15:47:42.548 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.14 2019-12-21T15:47:42.548 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:42.548 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:42.548 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.14") | .last_scrub_stamp' 2019-12-21T15:47:42.548 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:42.548 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:42.549 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:42.549 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:42.549 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.14 last_scrub_stamp 2019-12-21T15:47:42.549 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.14 2019-12-21T15:47:42.549 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:42.550 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:42.550 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.14") | .last_scrub_stamp' 2019-12-21T15:47:42.550 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:42.550 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:42.551 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:42.551 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:42.551 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.14 last_scrub_stamp 2019-12-21T15:47:42.551 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.14 2019-12-21T15:47:42.551 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:42.657 INFO:tasks.workunit.client.0.smithi172.stderr:25i++ )) 2019-12-21T15:47:42.658 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:42.658 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.16 last_scrub_stamp 2019-12-21T15:47:42.658 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.16 2019-12-21T15:47:42.658 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:42.658 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:42.658 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.16") | .last_scrub_stamp' 2019-12-21T15:47:42.659 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:42.659 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:42.659 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:42.659 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:42.659 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.16 last_scrub_stamp 2019-12-21T15:47:42.659 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.16 2019-12-21T15:47:42.659 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:42.660 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.16") | .last_scrub_stamp' 2019-12-21T15:47:42.660 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:42.660 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:42.660 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:42.660 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:42.660 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:42.660 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.16 last_scrub_stamp 2019-12-21T15:47:42.661 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.16 2019-12-21T15:47:42.661 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:42.661 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:42.661 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.16") | .last_scrub_stamp' 2019-12-21T15:47:42.661 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:42.661 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:42.661 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:42.662 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:42.662 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.16 last_scrub_stamp 2019-12-21T15:47:42.662 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.16 2019-12-21T15:47:42.662 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:42.741 INFO:tasks.workunit.client.0.smithi172.stderr:25i++ )) 2019-12-21T15:47:42.794 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:42.794 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.1b last_scrub_stamp 2019-12-21T15:47:42.794 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.1b 2019-12-21T15:47:42.794 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:42.795 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:42.795 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1b") | .last_scrub_stamp' 2019-12-21T15:47:42.795 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:42.795 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:42.795 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:42.795 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:42.796 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.1b last_scrub_stamp 2019-12-21T15:47:42.796 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.1b 2019-12-21T15:47:42.796 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:42.796 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:42.796 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1b") | .last_scrub_stamp' 2019-12-21T15:47:42.796 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:42.797 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:42.797 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:42.797 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:42.797 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.1b last_scrub_stamp 2019-12-21T15:47:42.797 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.1b 2019-12-21T15:47:42.797 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:42.798 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1b") | .last_scrub_stamp' 2019-12-21T15:47:42.798 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:42.798 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:42.798 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:42.798 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:42.798 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:42.798 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.1b last_scrub_stamp 2019-12-21T15:47:42.799 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.1b 2019-12-21T15:47:42.799 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:42.808 INFO:tasks.workunit.client.0.smithi172.stderr:25i++ )) 2019-12-21T15:47:42.809 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:42.809 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.15 last_scrub_stamp 2019-12-21T15:47:42.809 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.15 2019-12-21T15:47:42.809 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:42.809 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:42.809 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.15") | .last_scrub_stamp' 2019-12-21T15:47:42.809 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:42.810 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:42.810 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:42.810 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:42.810 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.15 last_scrub_stamp 2019-12-21T15:47:42.810 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.15 2019-12-21T15:47:42.810 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:42.810 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:42.811 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.15") | .last_scrub_stamp' 2019-12-21T15:47:42.811 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:42.811 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:42.811 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:42.811 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:42.811 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.15 last_scrub_stamp 2019-12-21T15:47:42.812 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.15 2019-12-21T15:47:42.812 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:42.812 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:42.812 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.15") | .last_scrub_stamp' 2019-12-21T15:47:42.812 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:42.812 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:42.812 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:42.813 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:42.813 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.15 last_scrub_stamp 2019-12-21T15:47:42.813 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.15 2019-12-21T15:47:42.813 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:45.877 INFO:tasks.workunit.client.0.smithi172.stderr:25st/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:45.877 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.4") | .last_scrub_stamp' 2019-12-21T15:47:45.878 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:45.878 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:45.878 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:45.878 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:45.878 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.4 last_scrub_stamp 2019-12-21T15:47:45.878 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.4 2019-12-21T15:47:45.879 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:45.879 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.4") | .last_scrub_stamp' 2019-12-21T15:47:45.879 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:45.879 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:45.879 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:45.879 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:45.879 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:45.880 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.4 last_scrub_stamp 2019-12-21T15:47:45.880 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.4 2019-12-21T15:47:45.880 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:45.880 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:45.880 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.4") | .last_scrub_stamp' 2019-12-21T15:47:45.881 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:43.276330+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:45.881 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1795: wait_for_scrub: return 0 2019-12-21T15:47:45.881 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1911: run_in_background: return 0 2019-12-21T15:47:46.245 INFO:tasks.workunit.client.0.smithi172.stderr:st/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:46.245 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.7") | .last_scrub_stamp' 2019-12-21T15:47:46.245 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:46.245 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:46.246 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:46.246 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:46.246 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.7 last_scrub_stamp 2019-12-21T15:47:46.246 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.7 2019-12-21T15:47:46.246 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:46.246 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.7") | .last_scrub_stamp' 2019-12-21T15:47:46.246 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:46.247 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:46.247 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:46.247 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:46.247 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:46.247 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.7 last_scrub_stamp 2019-12-21T15:47:46.247 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.7 2019-12-21T15:47:46.247 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:46.248 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.7") | .last_scrub_stamp' 2019-12-21T15:47:46.248 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:46.248 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:43.144459+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:46.248 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1795: wait_for_scrub: return 0 2019-12-21T15:47:46.248 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1911: run_in_background: return 0 2019-12-21T15:47:47.080 INFO:tasks.workunit.client.0.smithi172.stderr:st/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.5") | .last_scrub_stamp' 2019-12-21T15:47:47.080 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:47.080 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:47.081 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:47.081 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:47.081 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:47.081 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.5 last_scrub_stamp 2019-12-21T15:47:47.081 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.5 2019-12-21T15:47:47.081 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:47.081 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:47.082 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.5") | .last_scrub_stamp' 2019-12-21T15:47:47.082 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:47.082 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:47.082 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:47.082 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:47.082 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.5 last_scrub_stamp 2019-12-21T15:47:47.082 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.5 2019-12-21T15:47:47.083 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:47.083 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:47.083 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.5") | .last_scrub_stamp' 2019-12-21T15:47:47.083 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:47.083 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:47.083 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:47.084 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:47.084 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.5 last_scrub_stamp 2019-12-21T15:47:47.084 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.5 2019-12-21T15:47:47.084 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:47.084 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.5") | .last_scrub_stamp' 2019-12-21T15:47:47.084 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:47.084 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:42.383714+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:47.085 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1795: wait_for_scrub: return 0 2019-12-21T15:47:47.085 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1911: run_in_background: return 0 2019-12-21T15:47:47.099 INFO:tasks.workunit.client.0.smithi172.stderr:st/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.2") | .last_scrub_stamp' 2019-12-21T15:47:47.100 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:47.100 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:47.100 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:47.100 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:47.100 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:47.101 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.2 last_scrub_stamp 2019-12-21T15:47:47.101 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.2 2019-12-21T15:47:47.101 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:47.101 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.2") | .last_scrub_stamp' 2019-12-21T15:47:47.101 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:47.101 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:47.102 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:47.102 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:47.102 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:47.102 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.2 last_scrub_stamp 2019-12-21T15:47:47.102 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.2 2019-12-21T15:47:47.102 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:47.102 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:47.103 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.2") | .last_scrub_stamp' 2019-12-21T15:47:47.103 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:47.103 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:47.103 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:47.103 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:47.103 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.2 last_scrub_stamp 2019-12-21T15:47:47.103 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.2 2019-12-21T15:47:47.104 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:47.104 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:47.104 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.2") | .last_scrub_stamp' 2019-12-21T15:47:47.104 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:47.242 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:st/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2019-12-21T15:47:47.243 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:47.243 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:47.243 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:47.243 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:47.243 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:47.243 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2019-12-21T15:47:47.243 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.0 2019-12-21T15:47:47.244 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:47.244 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2019-12-21T15:47:47.244 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:47.244 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:47.244 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:47.244 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:47.245 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:47.245 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2019-12-21T15:47:47.245 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.0 2019-12-21T15:47:47.245 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:47.245 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:47.245 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2019-12-21T15:47:47.245 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:47.246 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:47.246 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:47.246 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:47.246 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2019-12-21T15:47:47.246 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.0 2019-12-21T15:47:47.246 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:47.246 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2019-12-21T15:47:47.247 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:47.247 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:47.356 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:st/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:47.356 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.e") | .last_scrub_stamp' 2019-12-21T15:47:47.356 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:47.356 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:47.357 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:47.357 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:47.357 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.e last_scrub_stamp 2019-12-21T15:47:47.357 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.e 2019-12-21T15:47:47.357 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:47.357 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:47.357 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.e") | .last_scrub_stamp' 2019-12-21T15:47:47.358 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:47.358 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:47.358 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:47.358 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:47.358 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.e last_scrub_stamp 2019-12-21T15:47:47.358 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.e 2019-12-21T15:47:47.358 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:47.359 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:47.359 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.e") | .last_scrub_stamp' 2019-12-21T15:47:47.359 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:47.359 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:47.359 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:47.359 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:47.359 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.e last_scrub_stamp 2019-12-21T15:47:47.360 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.e 2019-12-21T15:47:47.360 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:47.360 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.e") | .last_scrub_stamp' 2019-12-21T15:47:47.360 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:47.360 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:47.465 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:47.466 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.13") | .last_scrub_stamp' 2019-12-21T15:47:47.466 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:47.466 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:47.466 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:47.466 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:47.466 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.13 last_scrub_stamp 2019-12-21T15:47:47.467 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.13 2019-12-21T15:47:47.467 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:47.467 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.13") | .last_scrub_stamp' 2019-12-21T15:47:47.467 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:47.467 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:47.467 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:47.467 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:47.468 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:47.468 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.13 last_scrub_stamp 2019-12-21T15:47:47.468 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.13 2019-12-21T15:47:47.468 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:47.468 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:47.468 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.13") | .last_scrub_stamp' 2019-12-21T15:47:47.468 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:47.469 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:47.469 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:47.469 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:47.469 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.13 last_scrub_stamp 2019-12-21T15:47:47.469 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.13 2019-12-21T15:47:47.469 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:47.469 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.13") | .last_scrub_stamp' 2019-12-21T15:47:47.470 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:47.470 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:47.494 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.cliest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:47.494 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.9") | .last_scrub_stamp' 2019-12-21T15:47:47.494 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:47.494 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:47.494 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:47.495 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:47.495 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.9 last_scrub_stamp 2019-12-21T15:47:47.495 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.9 2019-12-21T15:47:47.495 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:47.495 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:47.495 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.9") | .last_scrub_stamp' 2019-12-21T15:47:47.495 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:47.496 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:47.496 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:47.496 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:47.496 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.9 last_scrub_stamp 2019-12-21T15:47:47.496 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.9 2019-12-21T15:47:47.496 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:47.496 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:47.497 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.9") | .last_scrub_stamp' 2019-12-21T15:47:47.497 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:47.497 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:47.497 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:47.497 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:47.497 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.9 last_scrub_stamp 2019-12-21T15:47:47.497 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.9 2019-12-21T15:47:47.498 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:47.498 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.9") | .last_scrub_stamp' 2019-12-21T15:47:47.498 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:47.498 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:45.007638+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:47.498 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1795: wait_for_scrub: return 0 2019-12-21T15:47:47.498 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1911: run_in_background: return 0 2019-12-21T15:47:47.550 INFO:tasks.workunit.client.0.smithi172.stderr:st/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:47.550 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.c") | .last_scrub_stamp' 2019-12-21T15:47:47.550 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:47.551 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:47.551 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:47.551 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:47.551 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.c last_scrub_stamp 2019-12-21T15:47:47.551 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.c 2019-12-21T15:47:47.551 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:47.551 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:47.552 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.c") | .last_scrub_stamp' 2019-12-21T15:47:47.552 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:47.552 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:47.552 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:47.552 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:47.552 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.c last_scrub_stamp 2019-12-21T15:47:47.553 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.c 2019-12-21T15:47:47.553 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:47.553 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:47.553 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.c") | .last_scrub_stamp' 2019-12-21T15:47:47.553 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:47.553 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:47.553 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:47.554 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:47.554 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.c last_scrub_stamp 2019-12-21T15:47:47.554 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.c 2019-12-21T15:47:47.554 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:47.554 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:47.554 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.c") | .last_scrub_stamp' 2019-12-21T15:47:47.554 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:47.685 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:st/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:47.685 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1") | .last_scrub_stamp' 2019-12-21T15:47:47.686 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:47.686 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:47.686 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:47.686 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:47.686 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.1 last_scrub_stamp 2019-12-21T15:47:47.686 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.1 2019-12-21T15:47:47.687 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:47.687 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:47.687 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1") | .last_scrub_stamp' 2019-12-21T15:47:47.687 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:47.687 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:47.687 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:47.688 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:47.688 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.1 last_scrub_stamp 2019-12-21T15:47:47.688 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.1 2019-12-21T15:47:47.688 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:47.688 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:47.688 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1") | .last_scrub_stamp' 2019-12-21T15:47:47.688 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:47.689 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:47.689 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:47.689 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:47.689 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.1 last_scrub_stamp 2019-12-21T15:47:47.689 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.1 2019-12-21T15:47:47.689 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:47.689 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:47.690 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1") | .last_scrub_stamp' 2019-12-21T15:47:47.690 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:45.825726+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:47.690 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1795: wait_for_scrub: return 0 2019-12-21T15:47:47.690 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1911: run_in_background: return 0 2019-12-21T15:47:47.775 INFO:tasks.workunit.client.0.smithi172.stderr:813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:47.776 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.10") | .last_scrub_stamp' 2019-12-21T15:47:47.776 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:47.776 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:47.776 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:47.776 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:47.776 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.10 last_scrub_stamp 2019-12-21T15:47:47.777 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.10 2019-12-21T15:47:47.777 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:47.777 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.10") | .last_scrub_stamp' 2019-12-21T15:47:47.777 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:47.777 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:47.777 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:47.777 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:47.778 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:47.778 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.10 last_scrub_stamp 2019-12-21T15:47:47.778 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.10 2019-12-21T15:47:47.778 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:47.778 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:47.778 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.10") | .last_scrub_stamp' 2019-12-21T15:47:47.778 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:47.779 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:47.779 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:47.779 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:47.779 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.10 last_scrub_stamp 2019-12-21T15:47:47.779 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.10 2019-12-21T15:47:47.779 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:47.779 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:47.780 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.10") | .last_scrub_stamp' 2019-12-21T15:47:47.780 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:47.780 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.cliest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:47.780 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.b") | .last_scrub_stamp' 2019-12-21T15:47:47.781 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:47.781 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:47.781 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:47.781 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:47.781 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.b last_scrub_stamp 2019-12-21T15:47:47.781 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.b 2019-12-21T15:47:47.781 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:47.782 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:47.782 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.b") | .last_scrub_stamp' 2019-12-21T15:47:47.782 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:47.782 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:47.782 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:47.782 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:47.782 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.b last_scrub_stamp 2019-12-21T15:47:47.783 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.b 2019-12-21T15:47:47.783 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:47.783 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:47.783 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.b") | .last_scrub_stamp' 2019-12-21T15:47:47.783 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:47.783 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:47.783 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:47.784 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:47.784 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.b last_scrub_stamp 2019-12-21T15:47:47.784 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.b 2019-12-21T15:47:47.784 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:47.784 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:47.784 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.b") | .last_scrub_stamp' 2019-12-21T15:47:47.784 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:47.837 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.17") | .last_scrub_stamp' 2019-12-21T15:47:47.837 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:47.837 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:47.838 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:47.838 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:47.838 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:47.838 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.17 last_scrub_stamp 2019-12-21T15:47:47.838 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.17 2019-12-21T15:47:47.838 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:47.839 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:47.839 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.17") | .last_scrub_stamp' 2019-12-21T15:47:47.839 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:47.839 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:47.839 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:47.839 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:47.839 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.17 last_scrub_stamp 2019-12-21T15:47:47.840 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.17 2019-12-21T15:47:47.840 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:47.840 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:47.840 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.17") | .last_scrub_stamp' 2019-12-21T15:47:47.840 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:47.840 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:47.840 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:47.841 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:47.841 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.17 last_scrub_stamp 2019-12-21T15:47:47.841 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.17 2019-12-21T15:47:47.841 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:47.841 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:47.841 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.17") | .last_scrub_stamp' 2019-12-21T15:47:47.841 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:47.946 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.cliest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.8") | .last_scrub_stamp' 2019-12-21T15:47:47.947 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:47.947 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:47.947 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:47.947 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:47.947 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:47.947 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.8 last_scrub_stamp 2019-12-21T15:47:47.948 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.8 2019-12-21T15:47:47.948 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:47.948 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:47.948 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.8") | .last_scrub_stamp' 2019-12-21T15:47:47.948 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:47.948 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:47.949 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:47.949 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:47.949 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.8 last_scrub_stamp 2019-12-21T15:47:47.949 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.8 2019-12-21T15:47:47.949 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:47.949 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:47.949 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.8") | .last_scrub_stamp' 2019-12-21T15:47:47.950 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:47.950 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:47.950 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:47.950 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:47.950 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.8 last_scrub_stamp 2019-12-21T15:47:47.950 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.8 2019-12-21T15:47:47.950 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:47.951 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:47.951 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.8") | .last_scrub_stamp' 2019-12-21T15:47:47.951 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:47.963 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:st/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:47.963 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.f") | .last_scrub_stamp' 2019-12-21T15:47:47.964 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:47.964 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:47.964 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:47.964 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:47.964 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.f last_scrub_stamp 2019-12-21T15:47:47.964 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.f 2019-12-21T15:47:47.965 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:47.965 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:47.965 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.f") | .last_scrub_stamp' 2019-12-21T15:47:47.965 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:47.965 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:47.965 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:47.965 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:47.966 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.f last_scrub_stamp 2019-12-21T15:47:47.966 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.f 2019-12-21T15:47:47.966 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:47.966 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.f") | .last_scrub_stamp' 2019-12-21T15:47:47.966 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:47.966 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:47.966 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:47.967 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:47.967 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:47.967 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.f last_scrub_stamp 2019-12-21T15:47:47.967 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.f 2019-12-21T15:47:47.967 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:47.967 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:47.969 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.f") | .last_scrub_stamp' 2019-12-21T15:47:47.969 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:47.969 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:st/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:47.969 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.d") | .last_scrub_stamp' 2019-12-21T15:47:47.969 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:47.970 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:47.970 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:47.970 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:47.970 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.d last_scrub_stamp 2019-12-21T15:47:47.970 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.d 2019-12-21T15:47:47.970 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:47.977 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.d") | .last_scrub_stamp' 2019-12-21T15:47:47.977 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:47.977 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:47.978 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:47.978 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:47.978 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:47.978 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.d last_scrub_stamp 2019-12-21T15:47:47.978 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.d 2019-12-21T15:47:47.978 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:47.978 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.d") | .last_scrub_stamp' 2019-12-21T15:47:47.979 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:47.979 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:47.979 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:47.979 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:47.979 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:47.979 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.d last_scrub_stamp 2019-12-21T15:47:47.979 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.d 2019-12-21T15:47:47.980 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:47.980 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:47.980 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.d") | .last_scrub_stamp' 2019-12-21T15:47:47.980 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:48.043 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:48.043 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.12") | .last_scrub_stamp' 2019-12-21T15:47:48.044 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:48.044 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:48.044 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:48.044 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:48.044 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.12 last_scrub_stamp 2019-12-21T15:47:48.044 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.12 2019-12-21T15:47:48.044 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:48.045 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:48.045 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.12") | .last_scrub_stamp' 2019-12-21T15:47:48.045 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:48.045 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:48.045 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:48.045 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:48.045 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.12 last_scrub_stamp 2019-12-21T15:47:48.046 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.12 2019-12-21T15:47:48.046 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:48.046 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.12") | .last_scrub_stamp' 2019-12-21T15:47:48.046 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:48.046 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:48.046 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:48.047 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:48.047 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:48.047 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.12 last_scrub_stamp 2019-12-21T15:47:48.047 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.12 2019-12-21T15:47:48.047 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:48.047 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:48.047 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.12") | .last_scrub_stamp' 2019-12-21T15:47:48.048 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:48.124 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.clie813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.19") | .last_scrub_stamp' 2019-12-21T15:47:48.124 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:48.124 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:48.124 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:48.124 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:48.124 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:48.125 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.19 last_scrub_stamp 2019-12-21T15:47:48.125 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.19 2019-12-21T15:47:48.125 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:48.125 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:48.125 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.19") | .last_scrub_stamp' 2019-12-21T15:47:48.125 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:48.125 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:48.126 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:48.126 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:48.126 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.19 last_scrub_stamp 2019-12-21T15:47:48.126 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.19 2019-12-21T15:47:48.126 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:48.126 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.19") | .last_scrub_stamp' 2019-12-21T15:47:48.126 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:48.127 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:48.127 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:48.127 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:48.127 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:48.127 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.19 last_scrub_stamp 2019-12-21T15:47:48.127 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.19 2019-12-21T15:47:48.128 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:48.128 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:48.128 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.19") | .last_scrub_stamp' 2019-12-21T15:47:48.128 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:48.257 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.clie813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:48.257 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1d") | .last_scrub_stamp' 2019-12-21T15:47:48.257 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:48.258 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:48.258 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:48.258 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:48.258 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.1d last_scrub_stamp 2019-12-21T15:47:48.258 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.1d 2019-12-21T15:47:48.258 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:48.259 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1d") | .last_scrub_stamp' 2019-12-21T15:47:48.259 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:48.259 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:48.259 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:48.259 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:48.259 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:48.259 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.1d last_scrub_stamp 2019-12-21T15:47:48.260 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.1d 2019-12-21T15:47:48.260 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:48.260 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1d") | .last_scrub_stamp' 2019-12-21T15:47:48.260 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:48.260 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:48.260 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:48.260 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:48.261 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:48.261 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.1d last_scrub_stamp 2019-12-21T15:47:48.261 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.1d 2019-12-21T15:47:48.261 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:48.261 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:48.261 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1d") | .last_scrub_stamp' 2019-12-21T15:47:48.261 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:48.288 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.clie813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:48.288 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1c") | .last_scrub_stamp' 2019-12-21T15:47:48.288 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:48.288 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:48.289 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:48.289 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:48.289 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.1c last_scrub_stamp 2019-12-21T15:47:48.289 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.1c 2019-12-21T15:47:48.289 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:48.289 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1c") | .last_scrub_stamp' 2019-12-21T15:47:48.289 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:48.290 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:48.290 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:48.290 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:48.290 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:48.290 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.1c last_scrub_stamp 2019-12-21T15:47:48.290 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.1c 2019-12-21T15:47:48.290 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:48.291 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:48.291 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1c") | .last_scrub_stamp' 2019-12-21T15:47:48.291 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:48.291 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:48.291 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:48.291 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:48.291 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.1c last_scrub_stamp 2019-12-21T15:47:48.292 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.1c 2019-12-21T15:47:48.292 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:48.292 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:48.292 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1c") | .last_scrub_stamp' 2019-12-21T15:47:48.292 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:48.323 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.clie813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:48.323 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.18") | .last_scrub_stamp' 2019-12-21T15:47:48.323 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:48.323 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:48.324 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:48.324 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:48.324 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.18 last_scrub_stamp 2019-12-21T15:47:48.324 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.18 2019-12-21T15:47:48.324 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:48.325 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:48.325 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.18") | .last_scrub_stamp' 2019-12-21T15:47:48.325 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:48.325 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:48.325 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:48.326 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:48.326 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.18 last_scrub_stamp 2019-12-21T15:47:48.326 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.18 2019-12-21T15:47:48.326 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:48.326 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:48.326 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.18") | .last_scrub_stamp' 2019-12-21T15:47:48.327 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:48.327 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:48.327 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:48.327 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:48.327 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.18 last_scrub_stamp 2019-12-21T15:47:48.328 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.18 2019-12-21T15:47:48.329 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:48.329 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:48.329 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.18") | .last_scrub_stamp' 2019-12-21T15:47:48.329 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:48.391 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.clie813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1a") | .last_scrub_stamp' 2019-12-21T15:47:48.391 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:48.391 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:48.392 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:48.392 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:48.392 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:48.392 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.1a last_scrub_stamp 2019-12-21T15:47:48.392 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.1a 2019-12-21T15:47:48.392 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:48.393 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:48.393 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1a") | .last_scrub_stamp' 2019-12-21T15:47:48.393 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:48.393 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:48.393 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:48.393 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:48.393 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.1a last_scrub_stamp 2019-12-21T15:47:48.394 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.1a 2019-12-21T15:47:48.394 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:48.394 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1a") | .last_scrub_stamp' 2019-12-21T15:47:48.394 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:48.394 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:48.394 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:48.394 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:48.395 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:48.395 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.1a last_scrub_stamp 2019-12-21T15:47:48.395 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.1a 2019-12-21T15:47:48.395 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:48.395 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1a") | .last_scrub_stamp' 2019-12-21T15:47:48.395 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:48.395 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:48.457 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.clie813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.11") | .last_scrub_stamp' 2019-12-21T15:47:48.457 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:48.457 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:48.457 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:48.457 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:48.457 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:48.458 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.11 last_scrub_stamp 2019-12-21T15:47:48.458 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.11 2019-12-21T15:47:48.458 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:48.458 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.11") | .last_scrub_stamp' 2019-12-21T15:47:48.458 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:48.459 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:48.459 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:48.459 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:48.459 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:48.459 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.11 last_scrub_stamp 2019-12-21T15:47:48.459 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.11 2019-12-21T15:47:48.459 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:48.460 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:48.460 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.11") | .last_scrub_stamp' 2019-12-21T15:47:48.460 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:48.460 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:48.460 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:48.460 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:48.460 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.11 last_scrub_stamp 2019-12-21T15:47:48.461 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.11 2019-12-21T15:47:48.461 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:48.461 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.11") | .last_scrub_stamp' 2019-12-21T15:47:48.461 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:48.461 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:48.484 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.cliest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:48.484 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.a") | .last_scrub_stamp' 2019-12-21T15:47:48.484 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:48.485 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:48.485 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:48.485 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:48.485 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.a last_scrub_stamp 2019-12-21T15:47:48.485 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.a 2019-12-21T15:47:48.485 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:48.485 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:48.486 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.a") | .last_scrub_stamp' 2019-12-21T15:47:48.486 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:48.486 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:48.486 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:48.486 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:48.486 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.a last_scrub_stamp 2019-12-21T15:47:48.486 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.a 2019-12-21T15:47:48.487 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:48.487 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.a") | .last_scrub_stamp' 2019-12-21T15:47:48.487 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:48.487 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:48.487 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:48.487 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:48.487 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:48.488 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.a last_scrub_stamp 2019-12-21T15:47:48.488 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.a 2019-12-21T15:47:48.488 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:48.488 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:48.488 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.a") | .last_scrub_stamp' 2019-12-21T15:47:48.488 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:48.493 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:48.493 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1e") | .last_scrub_stamp' 2019-12-21T15:47:48.493 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:48.493 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:48.493 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:48.494 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:48.494 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.1e last_scrub_stamp 2019-12-21T15:47:48.494 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.1e 2019-12-21T15:47:48.494 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:48.494 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:48.494 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1e") | .last_scrub_stamp' 2019-12-21T15:47:48.494 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:48.495 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:48.495 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:48.495 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:48.495 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.1e last_scrub_stamp 2019-12-21T15:47:48.495 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.1e 2019-12-21T15:47:48.495 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:48.495 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:48.496 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1e") | .last_scrub_stamp' 2019-12-21T15:47:48.496 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:48.496 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:48.496 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:48.496 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:48.496 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.1e last_scrub_stamp 2019-12-21T15:47:48.497 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.1e 2019-12-21T15:47:48.497 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:48.497 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1e") | .last_scrub_stamp' 2019-12-21T15:47:48.497 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:48.497 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:48.532 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.clie813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:48.532 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.14") | .last_scrub_stamp' 2019-12-21T15:47:48.532 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:48.532 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:48.533 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:48.533 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:48.533 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.14 last_scrub_stamp 2019-12-21T15:47:48.533 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.14 2019-12-21T15:47:48.533 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:48.534 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.14") | .last_scrub_stamp' 2019-12-21T15:47:48.534 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:48.534 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:48.534 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:48.534 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:48.535 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:48.535 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.14 last_scrub_stamp 2019-12-21T15:47:48.535 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.14 2019-12-21T15:47:48.535 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:48.535 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.14") | .last_scrub_stamp' 2019-12-21T15:47:48.535 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:48.535 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:48.536 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:48.536 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:48.536 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:48.536 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.14 last_scrub_stamp 2019-12-21T15:47:48.536 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.14 2019-12-21T15:47:48.536 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:48.536 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.14") | .last_scrub_stamp' 2019-12-21T15:47:48.537 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:48.537 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:48.616 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.clie813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1f") | .last_scrub_stamp' 2019-12-21T15:47:48.616 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:48.616 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:48.616 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:48.616 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:48.617 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:48.617 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.1f last_scrub_stamp 2019-12-21T15:47:48.617 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.1f 2019-12-21T15:47:48.617 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:48.617 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:48.617 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1f") | .last_scrub_stamp' 2019-12-21T15:47:48.617 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:48.618 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:48.618 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:48.618 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:48.618 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.1f last_scrub_stamp 2019-12-21T15:47:48.618 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.1f 2019-12-21T15:47:48.618 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:48.618 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1f") | .last_scrub_stamp' 2019-12-21T15:47:48.619 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:48.619 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:48.619 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:48.619 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:48.619 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:48.619 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.1f last_scrub_stamp 2019-12-21T15:47:48.619 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.1f 2019-12-21T15:47:48.620 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:48.620 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:48.620 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1f") | .last_scrub_stamp' 2019-12-21T15:47:48.620 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:48.665 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.clie813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.15") | .last_scrub_stamp' 2019-12-21T15:47:48.665 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:48.665 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:48.665 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:48.665 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:48.665 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:48.666 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.15 last_scrub_stamp 2019-12-21T15:47:48.666 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.15 2019-12-21T15:47:48.666 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:48.666 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.15") | .last_scrub_stamp' 2019-12-21T15:47:48.666 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:48.666 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:48.667 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:48.667 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:48.667 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:48.667 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.15 last_scrub_stamp 2019-12-21T15:47:48.667 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.15 2019-12-21T15:47:48.667 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:48.667 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:48.668 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.15") | .last_scrub_stamp' 2019-12-21T15:47:48.668 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:48.668 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:48.668 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:48.668 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:48.668 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.15 last_scrub_stamp 2019-12-21T15:47:48.668 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.15 2019-12-21T15:47:48.669 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:48.669 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:48.669 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.15") | .last_scrub_stamp' 2019-12-21T15:47:48.669 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:48.685 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.clie813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1b") | .last_scrub_stamp' 2019-12-21T15:47:48.685 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:48.685 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:48.686 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:48.686 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:48.686 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:48.686 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.1b last_scrub_stamp 2019-12-21T15:47:48.686 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.1b 2019-12-21T15:47:48.686 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:48.686 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1b") | .last_scrub_stamp' 2019-12-21T15:47:48.687 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:48.687 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:48.687 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:48.687 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:48.687 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:48.687 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.1b last_scrub_stamp 2019-12-21T15:47:48.688 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.1b 2019-12-21T15:47:48.688 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:48.688 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:48.688 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1b") | .last_scrub_stamp' 2019-12-21T15:47:48.688 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:48.688 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:48.689 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:48.689 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:48.690 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.1b last_scrub_stamp 2019-12-21T15:47:48.690 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.1b 2019-12-21T15:47:48.690 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:48.690 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:48.690 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1b") | .last_scrub_stamp' 2019-12-21T15:47:48.690 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:48.830 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.clie813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.16") | .last_scrub_stamp' 2019-12-21T15:47:48.830 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:48.830 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:48.830 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:48.830 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:48.830 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:48.831 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.16 last_scrub_stamp 2019-12-21T15:47:48.831 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.16 2019-12-21T15:47:48.831 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:48.831 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:48.831 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.16") | .last_scrub_stamp' 2019-12-21T15:47:48.831 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:48.832 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:48.832 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:48.832 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:48.832 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.16 last_scrub_stamp 2019-12-21T15:47:48.832 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.16 2019-12-21T15:47:48.832 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:48.832 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.16") | .last_scrub_stamp' 2019-12-21T15:47:48.833 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:48.833 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:48.833 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:48.833 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:48.833 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:48.833 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.16 last_scrub_stamp 2019-12-21T15:47:48.833 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.16 2019-12-21T15:47:48.834 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:48.834 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:48.834 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.16") | .last_scrub_stamp' 2019-12-21T15:47:48.834 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:48.948 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.clie1797: wait_for_scrub: sleep 1 2019-12-21T15:47:48.948 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:48.948 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:48.948 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.c last_scrub_stamp 2019-12-21T15:47:48.948 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.c 2019-12-21T15:47:48.948 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:48.949 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:48.949 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.c") | .last_scrub_stamp' 2019-12-21T15:47:48.949 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:44.325227+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:48.949 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1795: wait_for_scrub: return 0 2019-12-21T15:47:48.949 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1911: run_in_background: return 0 2019-12-21T15:47:49.288 INFO:tasks.workunit.client.0.smithi172.stderr:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:49.288 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:49.288 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:49.289 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.f last_scrub_stamp 2019-12-21T15:47:49.289 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.f 2019-12-21T15:47:49.289 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:49.289 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:49.289 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.f") | .last_scrub_stamp' 2019-12-21T15:47:49.289 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:44.108758+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:49.289 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1795: wait_for_scrub: return 0 2019-12-21T15:47:49.290 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1911: run_in_background: return 0 2019-12-21T15:47:49.466 INFO:tasks.workunit.client.0.smithi172.stderr:nt.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:49.466 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:49.466 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:49.466 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.19 last_scrub_stamp 2019-12-21T15:47:49.466 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.19 2019-12-21T15:47:49.466 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:49.467 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:49.467 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.19") | .last_scrub_stamp' 2019-12-21T15:47:49.467 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:48.322918+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:49.467 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1795: wait_for_scrub: return 0 2019-12-21T15:47:49.467 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1911: run_in_background: return 0 2019-12-21T15:47:49.979 INFO:tasks.workunit.client.0.smithi172.stderr:nt.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:49.979 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:49.980 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:49.980 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.11 last_scrub_stamp 2019-12-21T15:47:49.980 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.11 2019-12-21T15:47:49.980 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:49.980 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.11") | .last_scrub_stamp' 2019-12-21T15:47:49.980 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:49.981 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:46.796943+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:49.981 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1795: wait_for_scrub: return 0 2019-12-21T15:47:49.981 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1911: run_in_background: return 0 2019-12-21T15:47:50.108 INFO:tasks.workunit.client.0.smithi172.stderr:nt.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:50.108 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:50.108 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:50.108 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.14 last_scrub_stamp 2019-12-21T15:47:50.108 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.14 2019-12-21T15:47:50.109 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:50.109 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.14") | .last_scrub_stamp' 2019-12-21T15:47:50.109 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:50.109 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:44.590184+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:50.109 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1795: wait_for_scrub: return 0 2019-12-21T15:47:50.109 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1911: run_in_background: return 0 2019-12-21T15:47:50.318 INFO:tasks.workunit.client.0.smithi172.stderr:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:50.318 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:50.318 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:50.319 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.2 last_scrub_stamp 2019-12-21T15:47:50.319 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.2 2019-12-21T15:47:50.319 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:50.319 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.2") | .last_scrub_stamp' 2019-12-21T15:47:50.319 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:50.319 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:50.319 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:50.320 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:50.320 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:50.320 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.2 last_scrub_stamp 2019-12-21T15:47:50.320 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.2 2019-12-21T15:47:50.320 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:50.320 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:50.320 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.2") | .last_scrub_stamp' 2019-12-21T15:47:50.321 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:43.861637+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:50.321 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1795: wait_for_scrub: return 0 2019-12-21T15:47:50.321 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1911: run_in_background: return 0 2019-12-21T15:47:50.389 INFO:tasks.workunit.client.0.smithi172.stderr:nt.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:50.390 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:50.390 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:50.390 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.16 last_scrub_stamp 2019-12-21T15:47:50.390 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.16 2019-12-21T15:47:50.390 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:50.390 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:50.391 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.16") | .last_scrub_stamp' 2019-12-21T15:47:50.391 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:48.040224+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:50.391 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1795: wait_for_scrub: return 0 2019-12-21T15:47:50.391 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1911: run_in_background: return 0 2019-12-21T15:47:51.327 INFO:tasks.workunit.client.0.smithi172.stderr:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:51.327 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:51.327 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:51.327 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.a last_scrub_stamp 2019-12-21T15:47:51.328 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.a 2019-12-21T15:47:51.328 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:51.328 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:51.328 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.a") | .last_scrub_stamp' 2019-12-21T15:47:51.328 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:51.328 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:51.328 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:51.329 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:51.329 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.a last_scrub_stamp 2019-12-21T15:47:51.329 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.a 2019-12-21T15:47:51.329 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:51.329 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:51.329 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.a") | .last_scrub_stamp' 2019-12-21T15:47:51.329 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:49.895261+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:51.330 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1795: wait_for_scrub: return 0 2019-12-21T15:47:51.330 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1911: run_in_background: return 0 2019-12-21T15:47:51.712 INFO:tasks.workunit.client.0.smithi172.stderr:nt.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:51.712 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:51.712 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:51.712 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.13 last_scrub_stamp 2019-12-21T15:47:51.712 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.13 2019-12-21T15:47:51.712 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:51.713 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.13") | .last_scrub_stamp' 2019-12-21T15:47:51.713 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:51.713 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:51.713 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:51.713 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:51.713 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:51.713 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.13 last_scrub_stamp 2019-12-21T15:47:51.714 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.13 2019-12-21T15:47:51.714 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:51.714 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.13") | .last_scrub_stamp' 2019-12-21T15:47:51.714 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:51.714 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:51.714 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:51.715 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:51.715 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:51.715 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.13 last_scrub_stamp 2019-12-21T15:47:51.715 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.13 2019-12-21T15:47:51.715 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:51.715 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:51.715 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.13") | .last_scrub_stamp' 2019-12-21T15:47:51.716 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:48.359295+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:51.716 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1795: wait_for_scrub: return 0 2019-12-21T15:47:51.716 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1911: run_in_background: return 0 2019-12-21T15:47:51.853 INFO:tasks.workunit.client.0.smithi172.stderr:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:51.853 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:51.853 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:51.853 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2019-12-21T15:47:51.853 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.0 2019-12-21T15:47:51.854 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:51.854 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:51.854 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2019-12-21T15:47:51.854 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:51.854 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:51.854 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:51.854 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:51.855 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2019-12-21T15:47:51.855 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.0 2019-12-21T15:47:51.855 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:51.855 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2019-12-21T15:47:51.855 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:51.855 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:51.855 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:51.856 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:51.856 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:51.856 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2019-12-21T15:47:51.856 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.0 2019-12-21T15:47:51.856 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:51.856 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:51.856 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2019-12-21T15:47:51.857 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:46.618109+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:51.857 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1795: wait_for_scrub: return 0 2019-12-21T15:47:51.857 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1911: run_in_background: return 0 2019-12-21T15:47:51.857 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1946: wait_background: for pid in $pids 2019-12-21T15:47:51.857 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1947: wait_background: wait 29870 2019-12-21T15:47:51.858 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1946: wait_background: for pid in $pids 2019-12-21T15:47:51.858 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1947: wait_background: wait 29877 2019-12-21T15:47:51.858 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1946: wait_background: for pid in $pids 2019-12-21T15:47:51.858 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1947: wait_background: wait 29884 2019-12-21T15:47:51.858 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1946: wait_background: for pid in $pids 2019-12-21T15:47:51.858 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1947: wait_background: wait 29891 2019-12-21T15:47:51.858 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1946: wait_background: for pid in $pids 2019-12-21T15:47:51.859 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1947: wait_background: wait 29898 2019-12-21T15:47:51.859 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1946: wait_background: for pid in $pids 2019-12-21T15:47:51.859 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1947: wait_background: wait 29902 2019-12-21T15:47:51.859 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1946: wait_background: for pid in $pids 2019-12-21T15:47:51.859 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1947: wait_background: wait 29910 2019-12-21T15:47:51.859 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1946: wait_background: for pid in $pids 2019-12-21T15:47:51.859 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1947: wait_background: wait 29912 2019-12-21T15:47:52.077 INFO:tasks.workunit.client.0.smithi172.stderr:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:52.077 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:52.078 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:52.078 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.d last_scrub_stamp 2019-12-21T15:47:52.078 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.d 2019-12-21T15:47:52.078 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:52.078 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:52.078 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.d") | .last_scrub_stamp' 2019-12-21T15:47:52.079 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:52.079 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:52.079 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:52.079 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:52.079 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.d last_scrub_stamp 2019-12-21T15:47:52.079 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.d 2019-12-21T15:47:52.079 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:52.080 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:52.080 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.d") | .last_scrub_stamp' 2019-12-21T15:47:52.080 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:52.080 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:52.080 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:52.080 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:52.081 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.d last_scrub_stamp 2019-12-21T15:47:52.081 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.d 2019-12-21T15:47:52.081 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:52.081 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:52.081 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.d") | .last_scrub_stamp' 2019-12-21T15:47:52.081 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:48.644829+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:52.081 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1795: wait_for_scrub: return 0 2019-12-21T15:47:52.082 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1911: run_in_background: return 0 2019-12-21T15:47:52.835 INFO:tasks.workunit.client.0.smithi172.stderr:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:52.836 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:52.836 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:52.836 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.e last_scrub_stamp 2019-12-21T15:47:52.837 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.e 2019-12-21T15:47:52.838 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:52.838 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.e") | .last_scrub_stamp' 2019-12-21T15:47:52.838 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:52.838 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:52.838 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:52.838 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:52.839 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:52.839 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.e last_scrub_stamp 2019-12-21T15:47:52.839 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.e 2019-12-21T15:47:52.839 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:52.839 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.e") | .last_scrub_stamp' 2019-12-21T15:47:52.839 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:52.839 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:52.840 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:52.840 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:52.840 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:52.840 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.e last_scrub_stamp 2019-12-21T15:47:52.840 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.e 2019-12-21T15:47:52.840 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:52.840 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:52.841 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.e") | .last_scrub_stamp' 2019-12-21T15:47:52.841 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:52.841 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:52.841 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:52.841 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:52.841 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.e last_scrub_stamp 2019-12-21T15:47:52.841 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.e 2019-12-21T15:47:52.999 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/unt.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:52.999 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:52.999 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:52.999 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.10 last_scrub_stamp 2019-12-21T15:47:52.999 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.10 2019-12-21T15:47:53.000 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:53.000 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:53.000 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.10") | .last_scrub_stamp' 2019-12-21T15:47:53.000 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:53.000 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:53.000 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:53.002 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:53.003 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.10 last_scrub_stamp 2019-12-21T15:47:53.003 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.10 2019-12-21T15:47:53.003 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:53.004 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.10") | .last_scrub_stamp' 2019-12-21T15:47:53.004 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:53.004 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:53.004 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:53.005 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:53.005 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:53.005 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.10 last_scrub_stamp 2019-12-21T15:47:53.005 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.10 2019-12-21T15:47:53.005 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:53.005 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:53.005 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.10") | .last_scrub_stamp' 2019-12-21T15:47:53.006 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:53.006 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:53.006 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:53.006 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:53.006 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.10 last_scrub_stamp 2019-12-21T15:47:53.023 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last1797: wait_for_scrub: sleep 1 2019-12-21T15:47:53.023 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:53.023 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:53.023 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.8 last_scrub_stamp 2019-12-21T15:47:53.023 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.8 2019-12-21T15:47:53.024 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:53.024 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:53.024 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.8") | .last_scrub_stamp' 2019-12-21T15:47:53.024 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:53.024 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:53.024 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:53.025 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:53.025 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.8 last_scrub_stamp 2019-12-21T15:47:53.025 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.8 2019-12-21T15:47:53.025 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:53.026 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.8") | .last_scrub_stamp' 2019-12-21T15:47:53.026 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:53.027 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:53.027 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:53.027 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:53.027 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:53.027 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.8 last_scrub_stamp 2019-12-21T15:47:53.028 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.8 2019-12-21T15:47:53.028 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:53.028 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:53.028 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.8") | .last_scrub_stamp' 2019-12-21T15:47:53.029 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:53.029 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:53.029 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:53.029 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:53.029 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.8 last_scrub_stamp 2019-12-21T15:47:53.029 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.8 2019-12-21T15:47:53.033 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/u1797: wait_for_scrub: sleep 1 2019-12-21T15:47:53.033 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:53.034 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:53.034 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.b last_scrub_stamp 2019-12-21T15:47:53.034 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.b 2019-12-21T15:47:53.034 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:53.034 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:53.035 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.b") | .last_scrub_stamp' 2019-12-21T15:47:53.035 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:53.035 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:53.035 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:53.035 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:53.035 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.b last_scrub_stamp 2019-12-21T15:47:53.035 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.b 2019-12-21T15:47:53.036 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:53.036 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:53.036 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.b") | .last_scrub_stamp' 2019-12-21T15:47:53.036 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:53.036 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:53.036 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:53.036 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:53.037 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.b last_scrub_stamp 2019-12-21T15:47:53.037 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.b 2019-12-21T15:47:53.037 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:53.037 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:53.037 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.b") | .last_scrub_stamp' 2019-12-21T15:47:53.037 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:53.038 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:53.038 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:53.039 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:53.039 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.b last_scrub_stamp 2019-12-21T15:47:53.039 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.b 2019-12-21T15:47:53.078 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/unt.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:53.079 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:53.079 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:53.079 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.17 last_scrub_stamp 2019-12-21T15:47:53.079 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.17 2019-12-21T15:47:53.079 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:53.080 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:53.080 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.17") | .last_scrub_stamp' 2019-12-21T15:47:53.080 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:53.080 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:53.080 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:53.080 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:53.081 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.17 last_scrub_stamp 2019-12-21T15:47:53.081 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.17 2019-12-21T15:47:53.081 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:53.081 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.17") | .last_scrub_stamp' 2019-12-21T15:47:53.081 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:53.081 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:53.081 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:53.082 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:53.082 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:53.082 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.17 last_scrub_stamp 2019-12-21T15:47:53.082 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.17 2019-12-21T15:47:53.082 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:53.082 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:53.082 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.17") | .last_scrub_stamp' 2019-12-21T15:47:53.083 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:53.083 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:53.083 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:53.083 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:53.083 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.17 last_scrub_stamp 2019-12-21T15:47:53.153 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_lastnt.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:53.153 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:53.153 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:53.154 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.12 last_scrub_stamp 2019-12-21T15:47:53.154 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.12 2019-12-21T15:47:53.154 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:53.154 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:53.154 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.12") | .last_scrub_stamp' 2019-12-21T15:47:53.154 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:53.154 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:53.155 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:53.155 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:53.155 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.12 last_scrub_stamp 2019-12-21T15:47:53.155 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.12 2019-12-21T15:47:53.155 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:53.155 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.12") | .last_scrub_stamp' 2019-12-21T15:47:53.155 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:53.156 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:53.156 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:53.156 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:53.156 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:53.156 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.12 last_scrub_stamp 2019-12-21T15:47:53.156 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.12 2019-12-21T15:47:53.156 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:53.157 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:53.157 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.12") | .last_scrub_stamp' 2019-12-21T15:47:53.158 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:53.158 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:53.158 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:53.158 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:53.158 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.12 last_scrub_stamp 2019-12-21T15:47:53.292 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_lastnt.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:53.292 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:53.292 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:53.292 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.1c last_scrub_stamp 2019-12-21T15:47:53.293 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.1c 2019-12-21T15:47:53.293 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:53.293 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:53.293 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1c") | .last_scrub_stamp' 2019-12-21T15:47:53.293 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:53.294 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:53.294 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:53.294 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:53.294 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.1c last_scrub_stamp 2019-12-21T15:47:53.294 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.1c 2019-12-21T15:47:53.294 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:53.294 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:53.295 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1c") | .last_scrub_stamp' 2019-12-21T15:47:53.295 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:53.295 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:53.295 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:53.295 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:53.295 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.1c last_scrub_stamp 2019-12-21T15:47:53.295 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.1c 2019-12-21T15:47:53.296 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:53.296 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1c") | .last_scrub_stamp' 2019-12-21T15:47:53.296 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:53.296 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:53.296 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:53.296 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:53.296 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:53.297 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.1c last_scrub_stamp 2019-12-21T15:47:53.297 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_lastnt.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:53.297 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:53.298 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:53.298 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.1d last_scrub_stamp 2019-12-21T15:47:53.298 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.1d 2019-12-21T15:47:53.298 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:53.299 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1d") | .last_scrub_stamp' 2019-12-21T15:47:53.299 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:53.299 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:53.299 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:53.299 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:53.299 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:53.300 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.1d last_scrub_stamp 2019-12-21T15:47:53.300 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.1d 2019-12-21T15:47:53.300 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:53.300 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:53.300 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1d") | .last_scrub_stamp' 2019-12-21T15:47:53.300 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:53.301 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:53.301 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:53.301 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:53.301 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.1d last_scrub_stamp 2019-12-21T15:47:53.301 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.1d 2019-12-21T15:47:53.301 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:53.301 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:53.302 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1d") | .last_scrub_stamp' 2019-12-21T15:47:53.302 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:53.302 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:53.302 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:53.302 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:53.303 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.1d last_scrub_stamp 2019-12-21T15:47:53.322 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_lastbuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:53.322 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:53.322 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.8") | .last_scrub_stamp' 2019-12-21T15:47:53.322 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:50.695045+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:53.322 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1795: wait_for_scrub: return 0 2019-12-21T15:47:53.323 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1911: run_in_background: return 0 2019-12-21T15:47:53.323 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1946: wait_background: for pid in $pids 2019-12-21T15:47:53.323 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1947: wait_background: wait 29918 2019-12-21T15:47:53.323 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1946: wait_background: for pid in $pids 2019-12-21T15:47:53.323 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1947: wait_background: wait 29922 2019-12-21T15:47:53.323 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1946: wait_background: for pid in $pids 2019-12-21T15:47:53.323 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1947: wait_background: wait 29924 2019-12-21T15:47:53.374 INFO:tasks.workunit.client.0.smithi172.stderr:nt.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:53.374 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:53.375 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:53.375 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.18 last_scrub_stamp 2019-12-21T15:47:53.375 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.18 2019-12-21T15:47:53.375 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:53.375 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:53.375 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.18") | .last_scrub_stamp' 2019-12-21T15:47:53.375 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:53.376 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:53.376 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:53.378 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:53.378 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.18 last_scrub_stamp 2019-12-21T15:47:53.379 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.18 2019-12-21T15:47:53.379 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:53.379 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:53.379 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.18") | .last_scrub_stamp' 2019-12-21T15:47:53.382 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:53.382 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:53.382 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:53.383 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:53.384 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.18 last_scrub_stamp 2019-12-21T15:47:53.384 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.18 2019-12-21T15:47:53.384 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:53.384 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:53.384 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.18") | .last_scrub_stamp' 2019-12-21T15:47:53.384 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:53.385 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:53.385 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:53.385 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:53.385 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.18 last_scrub_stamp 2019-12-21T15:47:53.397 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_lastnt.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:53.397 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:53.398 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:53.398 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.1a last_scrub_stamp 2019-12-21T15:47:53.398 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.1a 2019-12-21T15:47:53.398 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:53.398 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:53.399 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1a") | .last_scrub_stamp' 2019-12-21T15:47:53.399 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:53.399 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:53.399 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:53.400 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:53.400 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.1a last_scrub_stamp 2019-12-21T15:47:53.400 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.1a 2019-12-21T15:47:53.400 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:53.400 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:53.400 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1a") | .last_scrub_stamp' 2019-12-21T15:47:53.401 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:53.401 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:53.401 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:53.401 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:53.401 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.1a last_scrub_stamp 2019-12-21T15:47:53.401 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.1a 2019-12-21T15:47:53.401 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:53.402 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:53.402 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1a") | .last_scrub_stamp' 2019-12-21T15:47:53.402 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:53.402 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:53.402 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:53.403 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:53.403 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.1a last_scrub_stamp 2019-12-21T15:47:53.653 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_lastnt.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:53.654 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:53.654 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:53.654 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.1e last_scrub_stamp 2019-12-21T15:47:53.654 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.1e 2019-12-21T15:47:53.654 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:53.654 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:53.655 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1e") | .last_scrub_stamp' 2019-12-21T15:47:53.655 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:53.655 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:53.655 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:53.655 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:53.655 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.1e last_scrub_stamp 2019-12-21T15:47:53.655 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.1e 2019-12-21T15:47:53.656 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:53.656 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:53.656 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1e") | .last_scrub_stamp' 2019-12-21T15:47:53.656 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:53.656 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:53.656 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:53.656 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:53.657 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.1e last_scrub_stamp 2019-12-21T15:47:53.657 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.1e 2019-12-21T15:47:53.657 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:53.657 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:53.657 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1e") | .last_scrub_stamp' 2019-12-21T15:47:53.657 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:53.657 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:53.658 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:53.658 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:53.658 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.1e last_scrub_stamp 2019-12-21T15:47:53.885 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_lastnt.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:53.885 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:53.885 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:53.886 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.15 last_scrub_stamp 2019-12-21T15:47:53.886 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.15 2019-12-21T15:47:53.886 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:53.886 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:53.886 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.15") | .last_scrub_stamp' 2019-12-21T15:47:53.886 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:53.886 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:53.887 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:53.887 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:53.887 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.15 last_scrub_stamp 2019-12-21T15:47:53.887 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.15 2019-12-21T15:47:53.887 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:53.887 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:53.888 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.15") | .last_scrub_stamp' 2019-12-21T15:47:53.888 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:53.888 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:53.888 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:53.888 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:53.888 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.15 last_scrub_stamp 2019-12-21T15:47:53.889 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.15 2019-12-21T15:47:53.889 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:53.889 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:53.889 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.15") | .last_scrub_stamp' 2019-12-21T15:47:53.889 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:53.889 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:53.889 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:53.890 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:53.890 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.15 last_scrub_stamp 2019-12-21T15:47:53.890 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_lastnt.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:53.890 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:53.891 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:53.891 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.1f last_scrub_stamp 2019-12-21T15:47:53.891 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.1f 2019-12-21T15:47:53.891 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:53.891 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:53.892 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1f") | .last_scrub_stamp' 2019-12-21T15:47:53.892 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:53.892 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:53.892 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:53.892 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:53.893 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.1f last_scrub_stamp 2019-12-21T15:47:53.893 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.1f 2019-12-21T15:47:53.893 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:53.893 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:53.894 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1f") | .last_scrub_stamp' 2019-12-21T15:47:53.894 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:53.894 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:53.894 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:53.894 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:53.894 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.1f last_scrub_stamp 2019-12-21T15:47:53.894 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.1f 2019-12-21T15:47:53.895 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:53.895 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:53.895 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1f") | .last_scrub_stamp' 2019-12-21T15:47:53.896 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:53.896 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:53.897 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:53.897 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:53.897 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.1f last_scrub_stamp 2019-12-21T15:47:53.968 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_lastnt.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:53.968 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:53.969 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:53.969 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.1b last_scrub_stamp 2019-12-21T15:47:53.969 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.1b 2019-12-21T15:47:53.969 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:53.969 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:53.969 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1b") | .last_scrub_stamp' 2019-12-21T15:47:53.970 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:53.970 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:53.970 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:53.970 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:53.970 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.1b last_scrub_stamp 2019-12-21T15:47:53.970 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.1b 2019-12-21T15:47:53.970 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:53.971 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:53.971 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1b") | .last_scrub_stamp' 2019-12-21T15:47:53.971 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:53.971 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:53.971 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:53.971 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:53.971 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.1b last_scrub_stamp 2019-12-21T15:47:53.972 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.1b 2019-12-21T15:47:53.972 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:53.972 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:53.972 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1b") | .last_scrub_stamp' 2019-12-21T15:47:53.972 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:53.972 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:53.972 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:53.973 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:53.973 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.1b last_scrub_stamp 2019-12-21T15:47:54.880 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.12 2019-12-21T15:47:54.880 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:54.880 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:54.880 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.12") | .last_scrub_stamp' 2019-12-21T15:47:54.881 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:54.881 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:54.881 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:54.881 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:54.881 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.12 last_scrub_stamp 2019-12-21T15:47:54.881 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.12 2019-12-21T15:47:54.881 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:54.882 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:54.882 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.12") | .last_scrub_stamp' 2019-12-21T15:47:54.882 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:48.867984+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:54.882 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1795: wait_for_scrub: return 0 2019-12-21T15:47:54.882 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1911: run_in_background: return 0 2019-12-21T15:47:55.129 INFO:tasks.workunit.client.0.smithi172.stderr:_scrub_stamp: local pgid=1.1e 2019-12-21T15:47:55.129 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:55.130 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:55.130 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1e") | .last_scrub_stamp' 2019-12-21T15:47:55.130 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:55.130 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:55.131 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:55.131 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:55.131 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.1e last_scrub_stamp 2019-12-21T15:47:55.131 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.1e 2019-12-21T15:47:55.131 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:55.132 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:55.132 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1e") | .last_scrub_stamp' 2019-12-21T15:47:55.132 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:49.282416+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:55.132 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1795: wait_for_scrub: return 0 2019-12-21T15:47:55.133 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1911: run_in_background: return 0 2019-12-21T15:47:56.855 INFO:tasks.workunit.client.0.smithi172.stderr:buntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:56.856 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:56.856 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.e") | .last_scrub_stamp' 2019-12-21T15:47:56.856 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:56.856 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:56.856 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:56.856 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:56.857 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.e last_scrub_stamp 2019-12-21T15:47:56.857 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.e 2019-12-21T15:47:56.857 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:56.857 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:56.857 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.e") | .last_scrub_stamp' 2019-12-21T15:47:56.857 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:56.857 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:56.857 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:56.858 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:56.858 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.e last_scrub_stamp 2019-12-21T15:47:56.858 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.e 2019-12-21T15:47:56.858 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:56.858 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:56.858 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.e") | .last_scrub_stamp' 2019-12-21T15:47:56.858 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:56.859 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:56.859 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:56.859 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:56.859 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.e last_scrub_stamp 2019-12-21T15:47:56.859 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.e 2019-12-21T15:47:56.859 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:56.859 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:56.860 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.e") | .last_scrub_stamp' 2019-12-21T15:47:56.860 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:51.846556+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:56.860 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1795: wait_for_scrub: return 0 2019-12-21T15:47:56.860 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1911: run_in_background: return 0 2019-12-21T15:47:57.053 INFO:tasks.workunit.client.0.smithi172.stderr:_scrub_stamp: local pgid=1.10 2019-12-21T15:47:57.053 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:57.053 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:57.053 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.10") | .last_scrub_stamp' 2019-12-21T15:47:57.053 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:57.054 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:57.054 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:57.054 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:57.054 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.10 last_scrub_stamp 2019-12-21T15:47:57.054 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.10 2019-12-21T15:47:57.054 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:57.055 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:57.055 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.10") | .last_scrub_stamp' 2019-12-21T15:47:57.055 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:57.055 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:57.055 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:57.055 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:57.056 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.10 last_scrub_stamp 2019-12-21T15:47:57.056 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.10 2019-12-21T15:47:57.056 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:57.056 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:57.056 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.10") | .last_scrub_stamp' 2019-12-21T15:47:57.056 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:57.056 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:57.057 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:57.057 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:57.057 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.10 last_scrub_stamp 2019-12-21T15:47:57.057 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.10 2019-12-21T15:47:57.057 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:57.057 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:57.057 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.10") | .last_scrub_stamp' 2019-12-21T15:47:57.058 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:51.728692+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:57.058 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1795: wait_for_scrub: return 0 2019-12-21T15:47:57.058 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1911: run_in_background: return 0 2019-12-21T15:47:57.127 INFO:tasks.workunit.client.0.smithi172.stderr:buntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:57.127 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:57.127 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.b") | .last_scrub_stamp' 2019-12-21T15:47:57.127 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:57.128 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:57.128 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:57.128 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:57.128 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.b last_scrub_stamp 2019-12-21T15:47:57.128 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.b 2019-12-21T15:47:57.128 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:57.129 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:57.129 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.b") | .last_scrub_stamp' 2019-12-21T15:47:57.129 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:57.129 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:57.129 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:57.129 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:57.129 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.b last_scrub_stamp 2019-12-21T15:47:57.129 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.b 2019-12-21T15:47:57.130 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:57.130 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:57.130 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.b") | .last_scrub_stamp' 2019-12-21T15:47:57.130 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:57.130 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:57.130 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:57.130 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:57.131 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.b last_scrub_stamp 2019-12-21T15:47:57.131 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.b 2019-12-21T15:47:57.131 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:57.131 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:57.131 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.b") | .last_scrub_stamp' 2019-12-21T15:47:57.131 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:50.865402+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:57.131 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1795: wait_for_scrub: return 0 2019-12-21T15:47:57.132 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1911: run_in_background: return 0 2019-12-21T15:47:57.132 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1946: wait_background: for pid in $pids 2019-12-21T15:47:57.132 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1947: wait_background: wait 29941 2019-12-21T15:47:57.132 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1946: wait_background: for pid in $pids 2019-12-21T15:47:57.132 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1947: wait_background: wait 29949 2019-12-21T15:47:57.132 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1946: wait_background: for pid in $pids 2019-12-21T15:47:57.132 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1947: wait_background: wait 29956 2019-12-21T15:47:57.133 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1946: wait_background: for pid in $pids 2019-12-21T15:47:57.133 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1947: wait_background: wait 29963 2019-12-21T15:47:57.133 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1946: wait_background: for pid in $pids 2019-12-21T15:47:57.133 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1947: wait_background: wait 29967 2019-12-21T15:47:57.133 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1946: wait_background: for pid in $pids 2019-12-21T15:47:57.133 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1947: wait_background: wait 29973 2019-12-21T15:47:57.134 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1946: wait_background: for pid in $pids 2019-12-21T15:47:57.134 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1947: wait_background: wait 29979 2019-12-21T15:47:57.134 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1946: wait_background: for pid in $pids 2019-12-21T15:47:57.134 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1947: wait_background: wait 29982 2019-12-21T15:47:57.134 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1946: wait_background: for pid in $pids 2019-12-21T15:47:57.134 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1947: wait_background: wait 29990 2019-12-21T15:47:57.134 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1946: wait_background: for pid in $pids 2019-12-21T15:47:57.135 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1947: wait_background: wait 30002 2019-12-21T15:47:57.159 INFO:tasks.workunit.client.0.smithi172.stderr:_scrub_stamp: local pgid=1.17 2019-12-21T15:47:57.159 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:57.159 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:57.160 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.17") | .last_scrub_stamp' 2019-12-21T15:47:57.160 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:57.161 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:57.161 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:57.161 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:57.161 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.17 last_scrub_stamp 2019-12-21T15:47:57.161 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.17 2019-12-21T15:47:57.161 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:57.163 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:57.163 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.17") | .last_scrub_stamp' 2019-12-21T15:47:57.163 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:57.163 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:57.163 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:57.163 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:57.164 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.17 last_scrub_stamp 2019-12-21T15:47:57.164 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.17 2019-12-21T15:47:57.164 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:57.164 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:57.164 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.17") | .last_scrub_stamp' 2019-12-21T15:47:57.164 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:57.165 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:57.165 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:57.165 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:57.166 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.17 last_scrub_stamp 2019-12-21T15:47:57.166 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.17 2019-12-21T15:47:57.166 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:57.166 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.17") | .last_scrub_stamp' 2019-12-21T15:47:57.166 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:57.438 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0_scrub_stamp: local pgid=1.1c 2019-12-21T15:47:57.439 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:57.439 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1c") | .last_scrub_stamp' 2019-12-21T15:47:57.439 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:57.439 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:57.439 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:57.439 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:57.447 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:57.447 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.1c last_scrub_stamp 2019-12-21T15:47:57.448 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.1c 2019-12-21T15:47:57.448 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:57.448 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:57.448 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1c") | .last_scrub_stamp' 2019-12-21T15:47:57.448 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:57.448 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:57.448 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:57.449 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:57.449 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.1c last_scrub_stamp 2019-12-21T15:47:57.449 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.1c 2019-12-21T15:47:57.449 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:57.449 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:57.449 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1c") | .last_scrub_stamp' 2019-12-21T15:47:57.449 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:57.450 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:57.450 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:57.450 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:57.450 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.1c last_scrub_stamp 2019-12-21T15:47:57.450 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.1c 2019-12-21T15:47:57.450 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:57.450 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:57.451 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1c") | .last_scrub_stamp' 2019-12-21T15:47:57.507 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0_scrub_stamp: local pgid=1.1d 2019-12-21T15:47:57.507 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:57.507 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:57.507 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1d") | .last_scrub_stamp' 2019-12-21T15:47:57.508 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:57.508 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:57.508 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:57.508 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:57.508 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.1d last_scrub_stamp 2019-12-21T15:47:57.508 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.1d 2019-12-21T15:47:57.509 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:57.509 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:57.509 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1d") | .last_scrub_stamp' 2019-12-21T15:47:57.509 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:57.509 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:57.509 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:57.509 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:57.510 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.1d last_scrub_stamp 2019-12-21T15:47:57.510 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.1d 2019-12-21T15:47:57.510 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:57.510 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:57.510 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1d") | .last_scrub_stamp' 2019-12-21T15:47:57.510 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:57.511 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:57.511 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:57.511 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:57.511 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.1d last_scrub_stamp 2019-12-21T15:47:57.511 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.1d 2019-12-21T15:47:57.511 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:57.512 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:57.512 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1d") | .last_scrub_stamp' 2019-12-21T15:47:57.549 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0_scrub_stamp: local pgid=1.18 2019-12-21T15:47:57.549 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:57.549 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.18") | .last_scrub_stamp' 2019-12-21T15:47:57.549 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:57.549 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:57.549 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:57.550 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:57.550 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:57.550 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.18 last_scrub_stamp 2019-12-21T15:47:57.550 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.18 2019-12-21T15:47:57.550 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:57.550 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:57.550 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.18") | .last_scrub_stamp' 2019-12-21T15:47:57.551 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:57.551 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:57.551 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:57.551 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:57.551 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.18 last_scrub_stamp 2019-12-21T15:47:57.551 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.18 2019-12-21T15:47:57.551 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:57.551 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:57.552 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.18") | .last_scrub_stamp' 2019-12-21T15:47:57.552 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:57.552 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:57.552 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:57.552 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:57.552 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.18 last_scrub_stamp 2019-12-21T15:47:57.553 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.18 2019-12-21T15:47:57.553 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:57.553 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:57.553 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.18") | .last_scrub_stamp' 2019-12-21T15:47:57.553 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0_scrub_stamp: local pgid=1.1a 2019-12-21T15:47:57.554 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:57.554 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:57.554 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1a") | .last_scrub_stamp' 2019-12-21T15:47:57.554 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:57.554 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:57.554 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:57.554 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:57.555 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.1a last_scrub_stamp 2019-12-21T15:47:57.555 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.1a 2019-12-21T15:47:57.555 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:57.555 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:57.555 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1a") | .last_scrub_stamp' 2019-12-21T15:47:57.555 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:57.555 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:57.556 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:57.556 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:57.556 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.1a last_scrub_stamp 2019-12-21T15:47:57.556 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.1a 2019-12-21T15:47:57.556 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:57.557 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:57.557 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1a") | .last_scrub_stamp' 2019-12-21T15:47:57.557 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:57.557 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:57.557 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:57.557 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:57.557 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.1a last_scrub_stamp 2019-12-21T15:47:57.558 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.1a 2019-12-21T15:47:57.558 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:57.558 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:57.558 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1a") | .last_scrub_stamp' 2019-12-21T15:47:57.838 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0_scrub_stamp: local pgid=1.1f 2019-12-21T15:47:57.838 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:57.838 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:57.839 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1f") | .last_scrub_stamp' 2019-12-21T15:47:57.839 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:57.839 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:57.839 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:57.839 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:57.839 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.1f last_scrub_stamp 2019-12-21T15:47:57.840 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.1f 2019-12-21T15:47:57.840 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:57.840 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:57.840 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1f") | .last_scrub_stamp' 2019-12-21T15:47:57.840 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:57.840 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:57.840 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:57.841 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:57.841 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.1f last_scrub_stamp 2019-12-21T15:47:57.841 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.1f 2019-12-21T15:47:57.841 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:57.841 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:57.841 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1f") | .last_scrub_stamp' 2019-12-21T15:47:57.841 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:57.842 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:57.842 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:57.842 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:57.842 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.1f last_scrub_stamp 2019-12-21T15:47:57.842 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.1f 2019-12-21T15:47:57.842 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:57.842 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:57.843 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1f") | .last_scrub_stamp' 2019-12-21T15:47:57.936 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0_scrub_stamp: local pgid=1.15 2019-12-21T15:47:57.937 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:57.937 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:57.937 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.15") | .last_scrub_stamp' 2019-12-21T15:47:57.937 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:57.937 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:57.937 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:57.937 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:57.938 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.15 last_scrub_stamp 2019-12-21T15:47:57.938 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.15 2019-12-21T15:47:57.938 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:57.938 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:57.938 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.15") | .last_scrub_stamp' 2019-12-21T15:47:57.938 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:57.938 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:57.939 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:57.939 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:57.939 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.15 last_scrub_stamp 2019-12-21T15:47:57.939 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.15 2019-12-21T15:47:57.939 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:57.939 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:57.939 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.15") | .last_scrub_stamp' 2019-12-21T15:47:57.940 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:57.940 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:57.940 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:57.940 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:57.940 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.15 last_scrub_stamp 2019-12-21T15:47:57.940 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.15 2019-12-21T15:47:57.940 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:57.941 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:57.941 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.15") | .last_scrub_stamp' 2019-12-21T15:47:57.996 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0_scrub_stamp: local pgid=1.1b 2019-12-21T15:47:57.996 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:57.996 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:57.996 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1b") | .last_scrub_stamp' 2019-12-21T15:47:57.997 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:57.997 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:57.997 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:57.997 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:57.997 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.1b last_scrub_stamp 2019-12-21T15:47:57.997 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.1b 2019-12-21T15:47:57.998 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:57.998 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:57.998 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1b") | .last_scrub_stamp' 2019-12-21T15:47:57.998 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:57.998 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:57.998 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:57.998 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:57.999 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.1b last_scrub_stamp 2019-12-21T15:47:57.999 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.1b 2019-12-21T15:47:57.999 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:57.999 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:57.999 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1b") | .last_scrub_stamp' 2019-12-21T15:47:57.999 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:57.999 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:58.000 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:58.000 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:58.000 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.1b last_scrub_stamp 2019-12-21T15:47:58.000 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.1b 2019-12-21T15:47:58.000 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:58.000 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:58.000 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1b") | .last_scrub_stamp' 2019-12-21T15:47:59.236 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:59.237 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:59.237 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:59.237 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:59.237 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.1b last_scrub_stamp 2019-12-21T15:47:59.237 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.1b 2019-12-21T15:47:59.237 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:59.237 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:59.238 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1b") | .last_scrub_stamp' 2019-12-21T15:47:59.238 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:55.868881+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:59.238 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1795: wait_for_scrub: return 0 2019-12-21T15:47:59.238 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1911: run_in_background: return 0 2019-12-21T15:47:59.608 INFO:tasks.workunit.client.0.smithi172.stderr:/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:59.609 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:59.609 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:59.609 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:59.609 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.17 last_scrub_stamp 2019-12-21T15:47:59.609 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.17 2019-12-21T15:47:59.609 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:59.610 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:59.610 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.17") | .last_scrub_stamp' 2019-12-21T15:47:59.610 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:59.610 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:47:59.610 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:47:59.610 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:47:59.611 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.17 last_scrub_stamp 2019-12-21T15:47:59.611 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.17 2019-12-21T15:47:59.611 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:47:59.611 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:47:59.611 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.17") | .last_scrub_stamp' 2019-12-21T15:47:59.611 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:54.891363+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:47:59.611 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1795: wait_for_scrub: return 0 2019-12-21T15:47:59.612 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1911: run_in_background: return 0 2019-12-21T15:48:01.387 INFO:tasks.workunit.client.0.smithi172.stderr:/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:48:01.387 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:48:01.387 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:48:01.387 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:48:01.388 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.1a last_scrub_stamp 2019-12-21T15:48:01.388 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.1a 2019-12-21T15:48:01.388 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:48:01.388 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:48:01.388 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1a") | .last_scrub_stamp' 2019-12-21T15:48:01.388 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:48:01.389 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:48:01.389 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:48:01.389 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:48:01.389 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.1a last_scrub_stamp 2019-12-21T15:48:01.389 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.1a 2019-12-21T15:48:01.389 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:48:01.389 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:48:01.389 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1a") | .last_scrub_stamp' 2019-12-21T15:48:01.390 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:48:01.390 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:48:01.390 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:48:01.390 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:48:01.390 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.1a last_scrub_stamp 2019-12-21T15:48:01.390 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.1a 2019-12-21T15:48:01.390 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:48:01.391 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:48:01.391 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1a") | .last_scrub_stamp' 2019-12-21T15:48:01.391 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:57.841429+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:48:01.391 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1795: wait_for_scrub: return 0 2019-12-21T15:48:01.391 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1911: run_in_background: return 0 2019-12-21T15:48:01.652 INFO:tasks.workunit.client.0.smithi172.stderr:/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:48:01.653 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:48:01.653 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:48:01.653 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:48:01.653 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.15 last_scrub_stamp 2019-12-21T15:48:01.654 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.15 2019-12-21T15:48:01.654 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:48:01.654 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:48:01.654 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.15") | .last_scrub_stamp' 2019-12-21T15:48:01.654 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:48:01.654 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:48:01.654 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:48:01.655 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:48:01.655 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.15 last_scrub_stamp 2019-12-21T15:48:01.655 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.15 2019-12-21T15:48:01.655 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:48:01.655 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:48:01.655 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.15") | .last_scrub_stamp' 2019-12-21T15:48:01.655 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:48:01.656 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:48:01.656 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:48:01.656 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:48:01.656 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.15 last_scrub_stamp 2019-12-21T15:48:01.656 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.15 2019-12-21T15:48:01.656 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:48:01.656 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:48:01.657 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.15") | .last_scrub_stamp' 2019-12-21T15:48:01.657 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:56.808816+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:48:01.657 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1795: wait_for_scrub: return 0 2019-12-21T15:48:01.657 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1911: run_in_background: return 0 2019-12-21T15:48:01.657 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1946: wait_background: for pid in $pids 2019-12-21T15:48:01.657 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1947: wait_background: wait 30012 2019-12-21T15:48:01.657 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1946: wait_background: for pid in $pids 2019-12-21T15:48:01.658 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1947: wait_background: wait 30017 2019-12-21T15:48:01.658 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1946: wait_background: for pid in $pids 2019-12-21T15:48:01.658 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1947: wait_background: wait 30024 2019-12-21T15:48:02.291 INFO:tasks.workunit.client.0.smithi172.stderr:/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:48:02.292 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:48:02.292 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:48:02.292 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:48:02.292 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.1c last_scrub_stamp 2019-12-21T15:48:02.292 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.1c 2019-12-21T15:48:02.293 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:48:02.293 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:48:02.293 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1c") | .last_scrub_stamp' 2019-12-21T15:48:02.293 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:48:02.293 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:48:02.294 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:48:02.294 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:48:02.294 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.1c last_scrub_stamp 2019-12-21T15:48:02.294 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.1c 2019-12-21T15:48:02.294 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:48:02.294 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:48:02.294 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1c") | .last_scrub_stamp' 2019-12-21T15:48:02.295 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:48:02.295 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:48:02.295 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:48:02.295 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:48:02.295 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.1c last_scrub_stamp 2019-12-21T15:48:02.295 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.1c 2019-12-21T15:48:02.295 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:48:02.296 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:48:02.296 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1c") | .last_scrub_stamp' 2019-12-21T15:48:02.296 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:48:02.296 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:48:02.296 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:48:02.296 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:48:02.297 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:48:02.297 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:48:02.297 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:48:02.297 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:48:02.297 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.1d last_scrub_stamp 2019-12-21T15:48:02.297 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.1d 2019-12-21T15:48:02.298 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:48:02.298 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:48:02.298 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1d") | .last_scrub_stamp' 2019-12-21T15:48:02.298 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:48:02.298 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:48:02.298 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:48:02.298 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:48:02.299 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.1d last_scrub_stamp 2019-12-21T15:48:02.299 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.1d 2019-12-21T15:48:02.299 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:48:02.299 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:48:02.299 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1d") | .last_scrub_stamp' 2019-12-21T15:48:02.299 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:48:02.299 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:48:02.300 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:48:02.300 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:48:02.300 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.1d last_scrub_stamp 2019-12-21T15:48:02.300 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.1d 2019-12-21T15:48:02.300 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:48:02.300 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:48:02.301 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1d") | .last_scrub_stamp' 2019-12-21T15:48:02.301 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:48:02.301 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:48:02.301 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:48:02.301 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:48:02.365 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:48:02.365 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:48:02.365 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:48:02.365 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:48:02.366 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.18 last_scrub_stamp 2019-12-21T15:48:02.366 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.18 2019-12-21T15:48:02.366 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:48:02.366 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:48:02.366 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.18") | .last_scrub_stamp' 2019-12-21T15:48:02.366 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:48:02.366 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:48:02.367 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:48:02.367 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:48:02.367 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.18 last_scrub_stamp 2019-12-21T15:48:02.367 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.18 2019-12-21T15:48:02.367 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:48:02.367 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:48:02.368 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.18") | .last_scrub_stamp' 2019-12-21T15:48:02.368 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:48:02.368 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:48:02.368 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:48:02.368 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:48:02.368 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.18 last_scrub_stamp 2019-12-21T15:48:02.368 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.18 2019-12-21T15:48:02.369 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:48:02.369 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:48:02.370 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.18") | .last_scrub_stamp' 2019-12-21T15:48:02.370 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:48:02.370 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:48:02.370 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:48:02.370 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:48:02.572 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:48:02.572 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:48:02.573 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:48:02.573 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:48:02.573 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.1f last_scrub_stamp 2019-12-21T15:48:02.573 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.1f 2019-12-21T15:48:02.573 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:48:02.574 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:48:02.574 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1f") | .last_scrub_stamp' 2019-12-21T15:48:02.574 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:48:02.575 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:48:02.575 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:48:02.575 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:48:02.575 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.1f last_scrub_stamp 2019-12-21T15:48:02.575 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.1f 2019-12-21T15:48:02.575 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:48:02.576 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:48:02.576 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1f") | .last_scrub_stamp' 2019-12-21T15:48:02.576 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:48:02.576 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:48:02.576 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:48:02.576 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:48:02.576 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.1f last_scrub_stamp 2019-12-21T15:48:02.577 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.1f 2019-12-21T15:48:02.577 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:48:02.577 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:48:02.577 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1f") | .last_scrub_stamp' 2019-12-21T15:48:02.577 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:48:02.577 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:48:02.577 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:48:02.578 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:48:03.870 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.18 last_scrub_stamp 2019-12-21T15:48:03.871 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.18 2019-12-21T15:48:03.871 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:48:03.871 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:48:03.871 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.18") | .last_scrub_stamp' 2019-12-21T15:48:03.871 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:48:03.871 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:48:03.872 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:48:03.872 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:48:03.872 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.18 last_scrub_stamp 2019-12-21T15:48:03.872 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.18 2019-12-21T15:48:03.872 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:48:03.872 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:48:03.872 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.18") | .last_scrub_stamp' 2019-12-21T15:48:03.873 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:56.768746+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:48:03.873 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1795: wait_for_scrub: return 0 2019-12-21T15:48:03.873 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1911: run_in_background: return 0 2019-12-21T15:48:03.873 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1946: wait_background: for pid in $pids 2019-12-21T15:48:03.873 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1947: wait_background: wait 30028 2019-12-21T15:48:03.873 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1946: wait_background: for pid in $pids 2019-12-21T15:48:03.873 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1947: wait_background: wait 30034 2019-12-21T15:48:03.874 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1946: wait_background: for pid in $pids 2019-12-21T15:48:03.874 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1947: wait_background: wait 30056 2019-12-21T15:48:03.874 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1946: wait_background: for pid in $pids 2019-12-21T15:48:03.874 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1947: wait_background: wait 30065 2019-12-21T15:48:04.028 INFO:tasks.workunit.client.0.smithi172.stderr:helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.1f last_scrub_stamp 2019-12-21T15:48:04.028 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.1f 2019-12-21T15:48:04.028 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:48:04.029 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:48:04.029 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1f") | .last_scrub_stamp' 2019-12-21T15:48:04.029 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:48:04.029 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:48:04.029 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:48:04.029 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:48:04.029 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.1f last_scrub_stamp 2019-12-21T15:48:04.030 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.1f 2019-12-21T15:48:04.030 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:48:04.030 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:48:04.030 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1f") | .last_scrub_stamp' 2019-12-21T15:48:04.030 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:59.714652+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:48:04.031 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1795: wait_for_scrub: return 0 2019-12-21T15:48:04.031 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1911: run_in_background: return 0 2019-12-21T15:48:05.065 INFO:tasks.workunit.client.0.smithi172.stderr:helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.1c last_scrub_stamp 2019-12-21T15:48:05.065 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.1c 2019-12-21T15:48:05.065 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:48:05.065 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:48:05.065 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1c") | .last_scrub_stamp' 2019-12-21T15:48:05.066 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:48:05.066 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:48:05.066 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:48:05.066 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:48:05.066 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.1c last_scrub_stamp 2019-12-21T15:48:05.066 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.1c 2019-12-21T15:48:05.067 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:48:05.067 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:48:05.067 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1c") | .last_scrub_stamp' 2019-12-21T15:48:05.067 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:48:05.067 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:48:05.067 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:48:05.067 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:48:05.067 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.1c last_scrub_stamp 2019-12-21T15:48:05.068 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.1c 2019-12-21T15:48:05.068 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:48:05.068 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:48:05.068 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1c") | .last_scrub_stamp' 2019-12-21T15:48:05.068 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:48:00.962132+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:48:05.068 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1795: wait_for_scrub: return 0 2019-12-21T15:48:05.069 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1911: run_in_background: return 0 2019-12-21T15:48:05.069 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1946: wait_background: for pid in $pids 2019-12-21T15:48:05.069 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1947: wait_background: wait 30081 2019-12-21T15:48:06.065 INFO:tasks.workunit.client.0.smithi172.stderr:helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.1d last_scrub_stamp 2019-12-21T15:48:06.065 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.1d 2019-12-21T15:48:06.065 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:48:06.065 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:48:06.065 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1d") | .last_scrub_stamp' 2019-12-21T15:48:06.066 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:48:06.066 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:48:06.066 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:48:06.066 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:48:06.066 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.1d last_scrub_stamp 2019-12-21T15:48:06.066 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.1d 2019-12-21T15:48:06.066 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:48:06.067 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:48:06.067 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1d") | .last_scrub_stamp' 2019-12-21T15:48:06.067 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:48:06.067 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:48:06.067 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:48:06.067 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:48:06.067 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.1d last_scrub_stamp 2019-12-21T15:48:06.068 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.1d 2019-12-21T15:48:06.068 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:48:06.068 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:48:06.068 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1d") | .last_scrub_stamp' 2019-12-21T15:48:06.069 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:48:06.069 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:48:06.069 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:48:06.069 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:48:06.069 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.1d last_scrub_stamp 2019-12-21T15:48:06.070 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.1d 2019-12-21T15:48:06.070 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:48:06.070 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:48:07.534 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1d") | .last_scrub_stamp' 2019-12-21T15:48:07.534 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:47:15.630468+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:48:07.535 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:48:07.535 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:48:07.535 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:48:07.535 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.1d last_scrub_stamp 2019-12-21T15:48:07.535 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.1d 2019-12-21T15:48:07.535 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:48:07.535 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:48:07.536 INFO:tasks.workunit.client.0.smithi172.stderr:25813: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1d") | .last_scrub_stamp' 2019-12-21T15:48:07.536 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:48:00.864858+0000 '>' 2019-12-21T15:47:15.630468+0000 2019-12-21T15:48:07.536 INFO:tasks.workunit.client.0.smithi172.stderr:25813: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1795: wait_for_scrub: return 0 2019-12-21T15:48:07.536 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1911: run_in_background: return 0 2019-12-21T15:48:07.536 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1946: wait_background: for pid in $pids 2019-12-21T15:48:07.536 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1947: wait_background: wait 30088 2019-12-21T15:48:07.537 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1946: wait_background: for pid in $pids 2019-12-21T15:48:07.537 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1947: wait_background: wait 30121 2019-12-21T15:48:07.537 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1954: wait_background: eval 'pids='\'''\''' 2019-12-21T15:48:07.537 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1954: wait_background: pids= 2019-12-21T15:48:07.537 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1956: wait_background: return 0 2019-12-21T15:48:07.537 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:73: TEST_recovery_scrub: return_code=0 2019-12-21T15:48:07.537 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:74: TEST_recovery_scrub: '[' 0 -ne 0 ']' 2019-12-21T15:48:07.538 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:76: TEST_recovery_scrub: ERRORS=0 2019-12-21T15:48:07.538 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:77: TEST_recovery_scrub: find td/osd-recovery-scrub 2019-12-21T15:48:07.538 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:77: TEST_recovery_scrub: grep '[^/]*\.pid' 2019-12-21T15:48:07.539 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:77: TEST_recovery_scrub: pidfile='td/osd-recovery-scrub/osd.1.pid 2019-12-21T15:48:07.539 INFO:tasks.workunit.client.0.smithi172.stderr:td/osd-recovery-scrub/osd.4.pid 2019-12-21T15:48:07.539 INFO:tasks.workunit.client.0.smithi172.stderr:td/osd-recovery-scrub/osd.2.pid 2019-12-21T15:48:07.540 INFO:tasks.workunit.client.0.smithi172.stderr:td/osd-recovery-scrub/mgr.x.pid 2019-12-21T15:48:07.540 INFO:tasks.workunit.client.0.smithi172.stderr:td/osd-recovery-scrub/osd.7.pid 2019-12-21T15:48:07.540 INFO:tasks.workunit.client.0.smithi172.stderr:td/osd-recovery-scrub/mon.a.pid 2019-12-21T15:48:07.540 INFO:tasks.workunit.client.0.smithi172.stderr:td/osd-recovery-scrub/osd.0.pid 2019-12-21T15:48:07.540 INFO:tasks.workunit.client.0.smithi172.stderr:td/osd-recovery-scrub/osd.5.pid 2019-12-21T15:48:07.540 INFO:tasks.workunit.client.0.smithi172.stderr:td/osd-recovery-scrub/osd.3.pid 2019-12-21T15:48:07.540 INFO:tasks.workunit.client.0.smithi172.stderr:td/osd-recovery-scrub/osd.6.pid' 2019-12-21T15:48:07.541 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:78: TEST_recovery_scrub: cat td/osd-recovery-scrub/osd.1.pid td/osd-recovery-scrub/osd.4.pid td/osd-recovery-scrub/osd.2.pid td/osd-recovery-scrub/mgr.x.pid td/osd-recovery-scrub/osd.7.pid td/osd-recovery-scrub/mon.a.pid td/osd-recovery-scrub/osd.0.pid td/osd-recovery-scrub/osd.5.pid td/osd-recovery-scrub/osd.3.pid td/osd-recovery-scrub/osd.6.pid 2019-12-21T15:48:07.541 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:78: TEST_recovery_scrub: pid='26566 2019-12-21T15:48:07.541 INFO:tasks.workunit.client.0.smithi172.stderr:27697 2019-12-21T15:48:07.541 INFO:tasks.workunit.client.0.smithi172.stderr:26943 2019-12-21T15:48:07.541 INFO:tasks.workunit.client.0.smithi172.stderr:25935 2019-12-21T15:48:07.541 INFO:tasks.workunit.client.0.smithi172.stderr:28801 2019-12-21T15:48:07.542 INFO:tasks.workunit.client.0.smithi172.stderr:25869 2019-12-21T15:48:07.542 INFO:tasks.workunit.client.0.smithi172.stderr:26162 2019-12-21T15:48:07.542 INFO:tasks.workunit.client.0.smithi172.stderr:28047 2019-12-21T15:48:07.542 INFO:tasks.workunit.client.0.smithi172.stderr:27320 2019-12-21T15:48:07.542 INFO:tasks.workunit.client.0.smithi172.stderr:28424' 2019-12-21T15:48:07.542 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:79: TEST_recovery_scrub: kill -0 26566 27697 26943 25935 28801 25869 26162 28047 27320 28424 2019-12-21T15:48:07.542 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:87: TEST_recovery_scrub: kill_daemons td/osd-recovery-scrub 2019-12-21T15:48:07.543 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2019-12-21T15:48:07.543 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2019-12-21T15:48:07.543 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2019-12-21T15:48:07.543 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2019-12-21T15:48:07.543 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2019-12-21T15:48:07.647 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2019-12-21T15:48:07.647 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:89: TEST_recovery_scrub: declare -a err_strings 2019-12-21T15:48:07.648 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:90: TEST_recovery_scrub: err_strings[0]='not scheduling scrubs due to active recovery' 2019-12-21T15:48:07.648 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:95: TEST_recovery_scrub: expr 8 - 1 2019-12-21T15:48:07.648 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:95: TEST_recovery_scrub: seq 0 7 2019-12-21T15:48:07.649 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:95: TEST_recovery_scrub: for osd in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:48:07.649 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:97: TEST_recovery_scrub: grep 'failed to reserve\|not scheduling scrubs' td/osd-recovery-scrub/osd.0.log 2019-12-21T15:48:07.655 INFO:tasks.workunit.client.0.smithi172.stdout:2019-12-21T15:47:36.306+0000 7fd805223700 20 osd.0 pg_epoch: 50 pg[1.4( empty local-lis/les=48/50 n=0 ec=41/41 lis/c=48/48 les/c/f=50/50/0 sis=48) [3,0,6,1] r=1 lpr=48 crt=0'0 mlcod 0'0 active mbc={}] handle_scrub_reserve_request: failed to reserve remotely 2019-12-21T15:48:07.655 INFO:tasks.workunit.client.0.smithi172.stdout:2019-12-21T15:47:36.878+0000 7fd81ea56700 20 osd.0 50 sched_scrub not scheduling scrubs due to active recovery 2019-12-21T15:48:07.655 INFO:tasks.workunit.client.0.smithi172.stdout:2019-12-21T15:47:38.896+0000 7fd81ea56700 20 osd.0 50 sched_scrub not scheduling scrubs due to active recovery 2019-12-21T15:48:07.656 INFO:tasks.workunit.client.0.smithi172.stdout:2019-12-21T15:47:39.317+0000 7fd805223700 20 osd.0 pg_epoch: 50 pg[1.4( empty local-lis/les=48/50 n=0 ec=41/41 lis/c=48/48 les/c/f=50/50/0 sis=48) [3,0,6,1] r=1 lpr=48 crt=0'0 mlcod 0'0 active mbc={}] handle_scrub_reserve_request: failed to reserve remotely 2019-12-21T15:48:07.656 INFO:tasks.workunit.client.0.smithi172.stdout:2019-12-21T15:47:39.584+0000 7fd807227700 20 osd.0 pg_epoch: 50 pg[1.14( empty local-lis/les=48/50 n=0 ec=41/41 lis/c=48/48 les/c/f=50/50/0 sis=48) [2,0,4,5] r=1 lpr=48 crt=0'0 mlcod 0'0 active mbc={}] handle_scrub_reserve_request: failed to reserve remotely 2019-12-21T15:48:07.656 INFO:tasks.workunit.client.0.smithi172.stdout:2019-12-21T15:47:40.304+0000 7fd805223700 20 osd.0 pg_epoch: 50 pg[1.4( empty local-lis/les=48/50 n=0 ec=41/41 lis/c=48/48 les/c/f=50/50/0 sis=48) [3,0,6,1] r=1 lpr=48 crt=0'0 mlcod 0'0 active mbc={}] handle_scrub_reserve_request: failed to reserve remotely 2019-12-21T15:48:07.656 INFO:tasks.workunit.client.0.smithi172.stdout:2019-12-21T15:47:40.861+0000 7fd81ea56700 20 osd.0 50 sched_scrub not scheduling scrubs due to active recovery 2019-12-21T15:48:07.656 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:95: TEST_recovery_scrub: for osd in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:48:07.656 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:97: TEST_recovery_scrub: grep 'failed to reserve\|not scheduling scrubs' td/osd-recovery-scrub/osd.1.log 2019-12-21T15:48:07.660 INFO:tasks.workunit.client.0.smithi172.stdout:2019-12-21T15:47:29.992+0000 7fe443290700 20 osd.1 50 sched_scrub not scheduling scrubs due to active recovery 2019-12-21T15:48:07.660 INFO:tasks.workunit.client.0.smithi172.stdout:2019-12-21T15:47:31.001+0000 7fe443290700 20 osd.1 50 sched_scrub not scheduling scrubs due to active recovery 2019-12-21T15:48:07.660 INFO:tasks.workunit.client.0.smithi172.stdout:2019-12-21T15:47:36.306+0000 7fe429a5d700 20 osd.1 pg_epoch: 50 pg[1.4( empty local-lis/les=48/50 n=0 ec=41/41 lis/c=48/48 les/c/f=50/50/0 sis=48) [3,0,6,1] r=3 lpr=48 crt=0'0 mlcod 0'0 active mbc={}] handle_scrub_reserve_request: failed to reserve remotely 2019-12-21T15:48:07.661 INFO:tasks.workunit.client.0.smithi172.stdout:2019-12-21T15:47:37.018+0000 7fe443290700 20 osd.1 50 sched_scrub not scheduling scrubs due to active recovery 2019-12-21T15:48:07.661 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:95: TEST_recovery_scrub: for osd in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:48:07.661 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:97: TEST_recovery_scrub: grep 'failed to reserve\|not scheduling scrubs' td/osd-recovery-scrub/osd.2.log 2019-12-21T15:48:07.664 INFO:tasks.workunit.client.0.smithi172.stdout:2019-12-21T15:47:37.583+0000 7fbed7ca4700 20 osd.2 50 sched_scrub not scheduling scrubs due to active recovery 2019-12-21T15:48:07.664 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:95: TEST_recovery_scrub: for osd in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:48:07.664 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:97: TEST_recovery_scrub: grep 'failed to reserve\|not scheduling scrubs' td/osd-recovery-scrub/osd.3.log 2019-12-21T15:48:07.668 INFO:tasks.workunit.client.0.smithi172.stdout:2019-12-21T15:47:31.402+0000 7f1fc4252700 20 osd.3 50 sched_scrub not scheduling scrubs due to active recovery 2019-12-21T15:48:07.668 INFO:tasks.workunit.client.0.smithi172.stdout:2019-12-21T15:47:32.847+0000 7f1faba21700 20 osd.3 pg_epoch: 50 pg[1.2( empty local-lis/les=48/50 n=0 ec=41/41 lis/c=48/48 les/c/f=50/50/0 sis=48) [0,7,1,3] r=3 lpr=48 crt=0'0 mlcod 0'0 active mbc={}] handle_scrub_reserve_request: failed to reserve remotely 2019-12-21T15:48:07.668 INFO:tasks.workunit.client.0.smithi172.stdout:2019-12-21T15:47:33.653+0000 7f1faca23700 20 osd.3 pg_epoch: 50 pg[1.0( empty local-lis/les=48/49 n=0 ec=41/41 lis/c=48/48 les/c/f=49/50/0 sis=48) [7,0,2,3] r=3 lpr=48 crt=0'0 mlcod 0'0 active mbc={}] handle_scrub_reserve_request: failed to reserve remotely 2019-12-21T15:48:07.669 INFO:tasks.workunit.client.0.smithi172.stdout:2019-12-21T15:47:33.844+0000 7f1faba21700 20 osd.3 pg_epoch: 50 pg[1.2( empty local-lis/les=48/50 n=0 ec=41/41 lis/c=48/48 les/c/f=50/50/0 sis=48) [0,7,1,3] r=3 lpr=48 crt=0'0 mlcod 0'0 active mbc={}] handle_scrub_reserve_request: failed to reserve remotely 2019-12-21T15:48:07.669 INFO:tasks.workunit.client.0.smithi172.stdout:2019-12-21T15:47:34.363+0000 7f1fc4252700 20 osd.3 50 sched_scrub not scheduling scrubs due to active recovery 2019-12-21T15:48:07.669 INFO:tasks.workunit.client.0.smithi172.stdout:2019-12-21T15:47:34.814+0000 7f1faba21700 20 osd.3 pg_epoch: 50 pg[1.2( empty local-lis/les=48/50 n=0 ec=41/41 lis/c=48/48 les/c/f=50/50/0 sis=48) [0,7,1,3] r=3 lpr=48 crt=0'0 mlcod 0'0 active mbc={}] handle_scrub_reserve_request: failed to reserve remotely 2019-12-21T15:48:07.669 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:95: TEST_recovery_scrub: for osd in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:48:07.669 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:97: TEST_recovery_scrub: grep 'failed to reserve\|not scheduling scrubs' td/osd-recovery-scrub/osd.4.log 2019-12-21T15:48:07.672 INFO:tasks.workunit.client.0.smithi172.stdout:2019-12-21T15:47:32.383+0000 7feeb8679700 20 osd.4 50 sched_scrub not scheduling scrubs due to active recovery 2019-12-21T15:48:07.672 INFO:tasks.workunit.client.0.smithi172.stdout:2019-12-21T15:47:33.422+0000 7feeb8679700 20 osd.4 50 sched_scrub not scheduling scrubs due to active recovery 2019-12-21T15:48:07.673 INFO:tasks.workunit.client.0.smithi172.stdout:2019-12-21T15:47:34.398+0000 7feeb8679700 20 osd.4 50 sched_scrub not scheduling scrubs due to active recovery 2019-12-21T15:48:07.673 INFO:tasks.workunit.client.0.smithi172.stdout:2019-12-21T15:47:35.028+0000 7fee9ee46700 20 osd.4 pg_epoch: 50 pg[1.9( empty local-lis/les=48/50 n=0 ec=41/41 lis/c=48/48 les/c/f=50/50/0 sis=48) [1,4,5,2] r=1 lpr=48 crt=0'0 mlcod 0'0 active mbc={}] handle_scrub_reserve_request: failed to reserve remotely 2019-12-21T15:48:07.673 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:95: TEST_recovery_scrub: for osd in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:48:07.673 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:97: TEST_recovery_scrub: grep 'failed to reserve\|not scheduling scrubs' td/osd-recovery-scrub/osd.5.log 2019-12-21T15:48:07.677 INFO:tasks.workunit.client.0.smithi172.stdout:2019-12-21T15:47:29.045+0000 7f5944265700 20 osd.5 50 sched_scrub not scheduling scrubs due to active recovery 2019-12-21T15:48:07.678 INFO:tasks.workunit.client.0.smithi172.stdout:2019-12-21T15:47:30.030+0000 7f5944265700 20 osd.5 50 sched_scrub not scheduling scrubs due to active recovery 2019-12-21T15:48:07.678 INFO:tasks.workunit.client.0.smithi172.stdout:2019-12-21T15:47:35.029+0000 7f592aa32700 20 osd.5 pg_epoch: 50 pg[1.9( empty local-lis/les=48/50 n=0 ec=41/41 lis/c=48/48 les/c/f=50/50/0 sis=48) [1,4,5,2] r=2 lpr=48 crt=0'0 mlcod 0'0 active mbc={}] handle_scrub_reserve_request: failed to reserve remotely 2019-12-21T15:48:07.678 INFO:tasks.workunit.client.0.smithi172.stdout:2019-12-21T15:47:36.883+0000 7f5944265700 20 osd.5 50 sched_scrub not scheduling scrubs due to active recovery 2019-12-21T15:48:07.678 INFO:tasks.workunit.client.0.smithi172.stdout:2019-12-21T15:47:38.864+0000 7f5944265700 20 osd.5 50 sched_scrub not scheduling scrubs due to active recovery 2019-12-21T15:48:07.678 INFO:tasks.workunit.client.0.smithi172.stdout:2019-12-21T15:47:39.584+0000 7f592ca36700 20 osd.5 pg_epoch: 50 pg[1.14( empty local-lis/les=48/50 n=0 ec=41/41 lis/c=48/48 les/c/f=50/50/0 sis=48) [2,0,4,5] r=3 lpr=48 crt=0'0 mlcod 0'0 active mbc={}] handle_scrub_reserve_request: failed to reserve remotely 2019-12-21T15:48:07.679 INFO:tasks.workunit.client.0.smithi172.stdout:2019-12-21T15:47:39.973+0000 7f592aa32700 20 osd.5 pg_epoch: 50 pg[1.9( empty local-lis/les=48/50 n=0 ec=41/41 lis/c=48/48 les/c/f=50/50/0 sis=48) [1,4,5,2] r=2 lpr=48 crt=0'0 mlcod 0'0 active mbc={}] handle_scrub_reserve_request: failed to reserve remotely 2019-12-21T15:48:07.679 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:95: TEST_recovery_scrub: for osd in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:48:07.679 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:97: TEST_recovery_scrub: grep 'failed to reserve\|not scheduling scrubs' td/osd-recovery-scrub/osd.6.log 2019-12-21T15:48:07.683 INFO:tasks.workunit.client.0.smithi172.stdout:2019-12-21T15:47:34.016+0000 7fa6ae925700 20 osd.6 50 sched_scrub not scheduling scrubs due to active recovery 2019-12-21T15:48:07.684 INFO:tasks.workunit.client.0.smithi172.stdout:2019-12-21T15:47:35.054+0000 7fa6ae925700 20 osd.6 50 sched_scrub not scheduling scrubs due to active recovery 2019-12-21T15:48:07.684 INFO:tasks.workunit.client.0.smithi172.stdout:2019-12-21T15:47:39.135+0000 7fa6ae925700 20 osd.6 50 sched_scrub not scheduling scrubs due to active recovery 2019-12-21T15:48:07.685 INFO:tasks.workunit.client.0.smithi172.stdout:2019-12-21T15:47:39.317+0000 7fa6950f2700 20 osd.6 pg_epoch: 50 pg[1.4( empty local-lis/les=48/50 n=0 ec=41/41 lis/c=48/48 les/c/f=50/50/0 sis=48) [3,0,6,1] r=2 lpr=48 crt=0'0 mlcod 0'0 active mbc={}] handle_scrub_reserve_request: failed to reserve remotely 2019-12-21T15:48:07.685 INFO:tasks.workunit.client.0.smithi172.stdout:2019-12-21T15:47:40.304+0000 7fa6950f2700 20 osd.6 pg_epoch: 50 pg[1.4( empty local-lis/les=48/50 n=0 ec=41/41 lis/c=48/48 les/c/f=50/50/0 sis=48) [3,0,6,1] r=2 lpr=48 crt=0'0 mlcod 0'0 active mbc={}] handle_scrub_reserve_request: failed to reserve remotely 2019-12-21T15:48:07.685 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:95: TEST_recovery_scrub: for osd in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:48:07.686 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:97: TEST_recovery_scrub: grep 'failed to reserve\|not scheduling scrubs' td/osd-recovery-scrub/osd.7.log 2019-12-21T15:48:07.687 INFO:tasks.workunit.client.0.smithi172.stdout:2019-12-21T15:47:40.656+0000 7fb8bd2e3700 20 osd.7 50 sched_scrub not scheduling scrubs due to active recovery 2019-12-21T15:48:07.687 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:99: TEST_recovery_scrub: for err_string in "${err_strings[@]}" 2019-12-21T15:48:07.687 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:101: TEST_recovery_scrub: found=false 2019-12-21T15:48:07.687 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:102: TEST_recovery_scrub: expr 8 - 1 2019-12-21T15:48:07.687 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:102: TEST_recovery_scrub: seq 0 7 2019-12-21T15:48:07.688 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:102: TEST_recovery_scrub: for osd in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:48:07.688 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:104: TEST_recovery_scrub: grep 'not scheduling scrubs due to active recovery' td/osd-recovery-scrub/osd.0.log 2019-12-21T15:48:07.688 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:106: TEST_recovery_scrub: found=true 2019-12-21T15:48:07.689 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:102: TEST_recovery_scrub: for osd in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:48:07.689 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:104: TEST_recovery_scrub: grep 'not scheduling scrubs due to active recovery' td/osd-recovery-scrub/osd.1.log 2019-12-21T15:48:07.690 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:106: TEST_recovery_scrub: found=true 2019-12-21T15:48:07.690 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:102: TEST_recovery_scrub: for osd in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:48:07.690 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:104: TEST_recovery_scrub: grep 'not scheduling scrubs due to active recovery' td/osd-recovery-scrub/osd.2.log 2019-12-21T15:48:07.691 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:106: TEST_recovery_scrub: found=true 2019-12-21T15:48:07.692 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:102: TEST_recovery_scrub: for osd in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:48:07.692 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:104: TEST_recovery_scrub: grep 'not scheduling scrubs due to active recovery' td/osd-recovery-scrub/osd.3.log 2019-12-21T15:48:07.706 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:106: TEST_recovery_scrub: found=true 2019-12-21T15:48:07.706 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:102: TEST_recovery_scrub: for osd in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:48:07.706 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:104: TEST_recovery_scrub: grep 'not scheduling scrubs due to active recovery' td/osd-recovery-scrub/osd.4.log 2019-12-21T15:48:07.706 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:106: TEST_recovery_scrub: found=true 2019-12-21T15:48:07.707 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:102: TEST_recovery_scrub: for osd in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:48:07.707 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:104: TEST_recovery_scrub: grep 'not scheduling scrubs due to active recovery' td/osd-recovery-scrub/osd.5.log 2019-12-21T15:48:07.707 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:106: TEST_recovery_scrub: found=true 2019-12-21T15:48:07.707 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:102: TEST_recovery_scrub: for osd in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:48:07.707 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:104: TEST_recovery_scrub: grep 'not scheduling scrubs due to active recovery' td/osd-recovery-scrub/osd.6.log 2019-12-21T15:48:07.707 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:106: TEST_recovery_scrub: found=true 2019-12-21T15:48:07.707 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:102: TEST_recovery_scrub: for osd in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:48:07.708 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:104: TEST_recovery_scrub: grep 'not scheduling scrubs due to active recovery' td/osd-recovery-scrub/osd.7.log 2019-12-21T15:48:07.708 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:106: TEST_recovery_scrub: found=true 2019-12-21T15:48:07.708 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:109: TEST_recovery_scrub: '[' true = false ']' 2019-12-21T15:48:07.708 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:115: TEST_recovery_scrub: teardown td/osd-recovery-scrub 2019-12-21T15:48:07.708 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-scrub 2019-12-21T15:48:07.708 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2019-12-21T15:48:07.708 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-scrub KILL 2019-12-21T15:48:07.709 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2019-12-21T15:48:07.709 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2019-12-21T15:48:07.709 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2019-12-21T15:48:07.709 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2019-12-21T15:48:07.709 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2019-12-21T15:48:07.709 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2019-12-21T15:48:07.711 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2019-12-21T15:48:07.711 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2019-12-21T15:48:07.711 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2019-12-21T15:48:07.711 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2019-12-21T15:48:07.711 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2019-12-21T15:48:07.711 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2019-12-21T15:48:07.712 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local 'pattern=|/usr/lib/systemd/systemd-coredump %P %u %g %s %t %c %h %e' 2019-12-21T15:48:07.712 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' '|' = '|' ']' 2019-12-21T15:48:07.712 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:177: teardown: pattern= 2019-12-21T15:48:07.712 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2019-12-21T15:48:07.712 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname '' 2019-12-21T15:48:07.712 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls . 2019-12-21T15:48:07.713 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2019-12-21T15:48:07.713 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-scrub 2019-12-21T15:48:07.733 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2019-12-21T15:48:07.733 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:48:07.733 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.25813 2019-12-21T15:48:07.733 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.25813 2019-12-21T15:48:07.735 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2019-12-21T15:48:07.735 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2019-12-21T15:48:07.735 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:117: TEST_recovery_scrub: '[' 0 '!=' 0 ']' 2019-12-21T15:48:07.735 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:123: TEST_recovery_scrub: echo 'TEST PASSED' 2019-12-21T15:48:07.736 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:124: TEST_recovery_scrub: return 0 2019-12-21T15:48:07.736 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2057: main: code=0 2019-12-21T15:48:07.736 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2061: main: teardown td/osd-recovery-scrub 0 2019-12-21T15:48:07.736 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-scrub 2019-12-21T15:48:07.736 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2019-12-21T15:48:07.736 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-scrub KILL 2019-12-21T15:48:07.736 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2019-12-21T15:48:07.737 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2019-12-21T15:48:07.737 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2019-12-21T15:48:07.737 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2019-12-21T15:48:07.737 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2019-12-21T15:48:07.737 INFO:tasks.workunit.client.0.smithi172.stdout:TEST PASSED 2019-12-21T15:48:07.738 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2019-12-21T15:48:07.738 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2019-12-21T15:48:07.738 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2019-12-21T15:48:07.739 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2019-12-21T15:48:07.739 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2019-12-21T15:48:07.739 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2019-12-21T15:48:07.739 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2019-12-21T15:48:07.740 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local 'pattern=|/usr/lib/systemd/systemd-coredump %P %u %g %s %t %c %h %e' 2019-12-21T15:48:07.740 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' '|' = '|' ']' 2019-12-21T15:48:07.740 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:177: teardown: pattern= 2019-12-21T15:48:07.740 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2019-12-21T15:48:07.741 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname '' 2019-12-21T15:48:07.741 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls . 2019-12-21T15:48:07.742 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2019-12-21T15:48:07.742 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-scrub 2019-12-21T15:48:07.743 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2019-12-21T15:48:07.743 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:48:07.743 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.25813 2019-12-21T15:48:07.743 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.25813 2019-12-21T15:48:07.744 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2019-12-21T15:48:07.744 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2019-12-21T15:48:07.744 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2062: main: return 0 2019-12-21T15:48:07.745 INFO:teuthology.orchestra.run:Running command with timeout 3600 2019-12-21T15:48:07.745 INFO:teuthology.orchestra.run.smithi172:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2019-12-21T15:48:07.819 INFO:tasks.workunit:Running workunit scrub/osd-scrub-dump.sh... 2019-12-21T15:48:07.819 INFO:teuthology.orchestra.run.smithi172:workunit test scrub/osd-scrub-dump.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=2b368c3feb5f798e3a142f2a972094a812d3d958 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh 2019-12-21T15:48:07.872 INFO:tasks.workunit.client.0.smithi172.stderr:stty: 'standard input': Inappropriate ioctl for device 2019-12-21T15:48:07.880 INFO:tasks.workunit.client.0.smithi172.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2019-12-21T15:48:07.880 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2050: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2019-12-21T15:48:07.880 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2050: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2019-12-21T15:48:07.880 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2051: main: export PYTHONWARNINGS=ignore 2019-12-21T15:48:07.880 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2051: main: PYTHONWARNINGS=ignore 2019-12-21T15:48:07.880 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2052: main: export CEPH_CONF=/dev/null 2019-12-21T15:48:07.881 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2052: main: CEPH_CONF=/dev/null 2019-12-21T15:48:07.881 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2053: main: unset CEPH_ARGS 2019-12-21T15:48:07.881 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2055: main: local code 2019-12-21T15:48:07.881 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2056: main: run td/osd-scrub-dump 2019-12-21T15:48:07.881 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:25: run: local dir=td/osd-scrub-dump 2019-12-21T15:48:07.881 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:26: run: shift 2019-12-21T15:48:07.881 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:27: run: local SLEEP=0 2019-12-21T15:48:07.882 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:28: run: local CHUNK_MAX=5 2019-12-21T15:48:07.882 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:30: run: export CEPH_MON=127.0.0.1:7184 2019-12-21T15:48:07.882 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:30: run: CEPH_MON=127.0.0.1:7184 2019-12-21T15:48:07.882 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:31: run: export CEPH_ARGS 2019-12-21T15:48:07.883 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:32: run: uuidgen 2019-12-21T15:48:07.883 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:32: run: CEPH_ARGS+='--fsid=19b44843-6158-4426-874e-bcacde9394a0 --auth-supported=none ' 2019-12-21T15:48:07.884 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:33: run: CEPH_ARGS+='--mon-host=127.0.0.1:7184 ' 2019-12-21T15:48:07.884 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:34: run: CEPH_ARGS+='--osd_max_scrubs=4 ' 2019-12-21T15:48:07.884 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:35: run: CEPH_ARGS+='--osd_scrub_sleep=0 ' 2019-12-21T15:48:07.884 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:36: run: CEPH_ARGS+='--osd_scrub_chunk_max=5 ' 2019-12-21T15:48:07.884 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:37: run: CEPH_ARGS+='--osd_scrub_sleep=2 ' 2019-12-21T15:48:07.884 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:38: run: CEPH_ARGS+='--osd_pool_default_size=3 ' 2019-12-21T15:48:07.884 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:40: run: export -n CEPH_CLI_TEST_DUP_COMMAND 2019-12-21T15:48:07.885 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:41: run: set 2019-12-21T15:48:07.885 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:41: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2019-12-21T15:48:07.887 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:41: run: local funcs=TEST_recover_unexpected 2019-12-21T15:48:07.887 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:42: run: for func in $funcs 2019-12-21T15:48:07.888 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:43: run: setup td/osd-scrub-dump 2019-12-21T15:48:07.888 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-dump 2019-12-21T15:48:07.888 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-dump 2019-12-21T15:48:07.888 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-dump 2019-12-21T15:48:07.888 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2019-12-21T15:48:07.888 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-dump KILL 2019-12-21T15:48:07.889 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2019-12-21T15:48:07.889 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2019-12-21T15:48:07.889 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2019-12-21T15:48:07.889 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2019-12-21T15:48:07.889 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2019-12-21T15:48:07.889 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2019-12-21T15:48:07.889 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2019-12-21T15:48:07.889 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2019-12-21T15:48:07.890 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2019-12-21T15:48:07.890 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2019-12-21T15:48:07.890 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2019-12-21T15:48:07.890 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2019-12-21T15:48:07.890 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local 'pattern=|/usr/lib/systemd/systemd-coredump %P %u %g %s %t %c %h %e' 2019-12-21T15:48:07.890 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' '|' = '|' ']' 2019-12-21T15:48:07.890 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:177: teardown: pattern= 2019-12-21T15:48:07.891 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2019-12-21T15:48:07.891 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname '' 2019-12-21T15:48:07.891 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls . 2019-12-21T15:48:07.894 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2019-12-21T15:48:07.894 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-dump 2019-12-21T15:48:07.894 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2019-12-21T15:48:07.894 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:48:07.895 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:48:07.895 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.43630 2019-12-21T15:48:07.895 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2019-12-21T15:48:07.895 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2019-12-21T15:48:07.895 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-dump 2019-12-21T15:48:07.895 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2019-12-21T15:48:07.895 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:48:07.896 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:48:07.896 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.43630 2019-12-21T15:48:07.896 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2019-12-21T15:48:07.896 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2019-12-21T15:48:07.896 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2019-12-21T15:48:07.897 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2019-12-21T15:48:07.897 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-dump 1' TERM HUP INT 2019-12-21T15:48:07.897 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:44: run: TEST_recover_unexpected td/osd-scrub-dump 2019-12-21T15:48:07.897 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:50: TEST_recover_unexpected: local dir=td/osd-scrub-dump 2019-12-21T15:48:07.898 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:51: TEST_recover_unexpected: shift 2019-12-21T15:48:07.898 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:52: TEST_recover_unexpected: local OSDS=6 2019-12-21T15:48:07.898 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:53: TEST_recover_unexpected: local PGS=16 2019-12-21T15:48:07.898 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:54: TEST_recover_unexpected: local POOLS=3 2019-12-21T15:48:07.898 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:55: TEST_recover_unexpected: local OBJS=1000 2019-12-21T15:48:07.898 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:57: TEST_recover_unexpected: run_mon td/osd-scrub-dump a 2019-12-21T15:48:07.898 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-dump 2019-12-21T15:48:07.898 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2019-12-21T15:48:07.899 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2019-12-21T15:48:07.899 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2019-12-21T15:48:07.899 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-dump/a 2019-12-21T15:48:07.899 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-dump/a --run-dir=td/osd-scrub-dump 2019-12-21T15:48:08.182 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2019-12-21T15:48:08.182 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2019-12-21T15:48:08.182 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2019-12-21T15:48:08.182 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2019-12-21T15:48:08.182 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:48:08.182 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:48:08.183 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.43630/$cluster-$name.asok' 2019-12-21T15:48:08.183 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-dump/a '--log-file=td/osd-scrub-dump/$name.log' '--admin-socket=/tmp/ceph-asok.43630/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-dump/log --run-dir=td/osd-scrub-dump '--pid-file=td/osd-scrub-dump/$name.pid' --mon-allow-pool-delete --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 2019-12-21T15:48:08.357 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: cat 2019-12-21T15:48:08.358 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: get_config mon a fsid 2019-12-21T15:48:08.359 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: get_config: local daemon=mon 2019-12-21T15:48:08.359 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: get_config: local id=a 2019-12-21T15:48:08.359 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local config=fsid 2019-12-21T15:48:08.359 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: jq -r .fsid 2019-12-21T15:48:08.359 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: get_asok_path mon.a 2019-12-21T15:48:08.359 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2019-12-21T15:48:08.359 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2019-12-21T15:48:08.360 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:48:08.360 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:48:08.360 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:48:08.360 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-mon.a.asok 2019-12-21T15:48:08.360 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: CEPH_ARGS= 2019-12-21T15:48:08.360 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: ceph --format json daemon /tmp/ceph-asok.43630/ceph-mon.a.asok config get fsid 2019-12-21T15:48:08.446 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: get_config mon a mon_host 2019-12-21T15:48:08.447 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: get_config: local daemon=mon 2019-12-21T15:48:08.447 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: get_config: local id=a 2019-12-21T15:48:08.447 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local config=mon_host 2019-12-21T15:48:08.447 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: jq -r .mon_host 2019-12-21T15:48:08.447 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: get_asok_path mon.a 2019-12-21T15:48:08.447 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2019-12-21T15:48:08.448 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2019-12-21T15:48:08.448 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:48:08.448 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:48:08.448 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:48:08.448 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-mon.a.asok 2019-12-21T15:48:08.448 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: CEPH_ARGS= 2019-12-21T15:48:08.448 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: ceph --format json daemon /tmp/ceph-asok.43630/ceph-mon.a.asok config get mon_host 2019-12-21T15:48:08.534 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:58: TEST_recover_unexpected: run_mgr td/osd-scrub-dump x 2019-12-21T15:48:08.535 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:551: run_mgr: local dir=td/osd-scrub-dump 2019-12-21T15:48:08.535 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: shift 2019-12-21T15:48:08.535 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local id=x 2019-12-21T15:48:08.535 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2019-12-21T15:48:08.535 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local data=td/osd-scrub-dump/x 2019-12-21T15:48:08.535 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:558: run_mgr: get_asok_path 2019-12-21T15:48:08.536 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2019-12-21T15:48:08.536 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2019-12-21T15:48:08.536 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2019-12-21T15:48:08.536 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:48:08.536 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:48:08.536 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.43630/$cluster-$name.asok' 2019-12-21T15:48:08.536 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:558: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2019-12-21T15:48:08.537 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:558: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-dump/x '--log-file=td/osd-scrub-dump/$name.log' '--admin-socket=/tmp/ceph-asok.43630/$cluster-$name.asok' --run-dir=td/osd-scrub-dump '--pid-file=td/osd-scrub-dump/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2019-12-21T15:48:09.159 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:59: TEST_recover_unexpected: expr 6 - 1 2019-12-21T15:48:09.160 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:59: TEST_recover_unexpected: seq 0 5 2019-12-21T15:48:09.161 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:59: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:48:09.161 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:61: TEST_recover_unexpected: run_osd td/osd-scrub-dump 0 2019-12-21T15:48:09.161 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local dir=td/osd-scrub-dump 2019-12-21T15:48:09.165 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: shift 2019-12-21T15:48:09.165 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local id=0 2019-12-21T15:48:09.165 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2019-12-21T15:48:09.165 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local osd_data=td/osd-scrub-dump/0 2019-12-21T15:48:09.166 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local 'ceph_args=--fsid=19b44843-6158-4426-874e-bcacde9394a0 --auth-supported=none --mon-host=127.0.0.1:7184 --osd_max_scrubs=4 --osd_scrub_sleep=0 --osd_scrub_chunk_max=5 --osd_scrub_sleep=2 --osd_pool_default_size=3 ' 2019-12-21T15:48:09.166 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2019-12-21T15:48:09.166 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-journal-size=100' 2019-12-21T15:48:09.166 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2019-12-21T15:48:09.166 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-data=td/osd-scrub-dump/0' 2019-12-21T15:48:09.166 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-dump/0/journal' 2019-12-21T15:48:09.166 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --chdir=' 2019-12-21T15:48:09.167 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+= 2019-12-21T15:48:09.167 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --run-dir=td/osd-scrub-dump' 2019-12-21T15:48:09.167 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: get_asok_path 2019-12-21T15:48:09.167 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2019-12-21T15:48:09.167 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2019-12-21T15:48:09.167 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2019-12-21T15:48:09.167 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:48:09.168 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:48:09.168 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.43630/$cluster-$name.asok' 2019-12-21T15:48:09.168 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.43630/$cluster-$name.asok' 2019-12-21T15:48:09.168 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --debug-osd=20' 2019-12-21T15:48:09.168 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-ms=1' 2019-12-21T15:48:09.168 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-monc=20' 2019-12-21T15:48:09.168 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --log-file=td/osd-scrub-dump/$name.log' 2019-12-21T15:48:09.169 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --pid-file=td/osd-scrub-dump/$name.pid' 2019-12-21T15:48:09.169 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2019-12-21T15:48:09.169 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2019-12-21T15:48:09.169 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2019-12-21T15:48:09.169 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' ' 2019-12-21T15:48:09.169 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+= 2019-12-21T15:48:09.169 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: mkdir -p td/osd-scrub-dump/0 2019-12-21T15:48:09.170 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: uuidgen 2019-12-21T15:48:09.170 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: local uuid=af8f9195-a7a7-447d-b86e-cc8117fcdf29 2019-12-21T15:48:09.172 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: echo 'add osd0 af8f9195-a7a7-447d-b86e-cc8117fcdf29' 2019-12-21T15:48:09.172 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: ceph-authtool --gen-print-key 2019-12-21T15:48:09.172 INFO:tasks.workunit.client.0.smithi172.stdout:add osd0 af8f9195-a7a7-447d-b86e-cc8117fcdf29 2019-12-21T15:48:09.180 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: OSD_SECRET=AQC5Pv5dlfCaChAAsm1G66on5W6q/99NmRUn5Q== 2019-12-21T15:48:09.180 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo '{"cephx_secret": "AQC5Pv5dlfCaChAAsm1G66on5W6q/99NmRUn5Q=="}' 2019-12-21T15:48:09.181 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph osd new af8f9195-a7a7-447d-b86e-cc8117fcdf29 -i td/osd-scrub-dump/0/new.json 2019-12-21T15:48:09.448 INFO:tasks.workunit.client.0.smithi172.stdout:0 2019-12-21T15:48:09.458 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: rm td/osd-scrub-dump/0/new.json 2019-12-21T15:48:09.458 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-osd -i 0 --fsid=19b44843-6158-4426-874e-bcacde9394a0 --auth-supported=none --mon-host=127.0.0.1:7184 --osd_max_scrubs=4 --osd_scrub_sleep=0 --osd_scrub_chunk_max=5 --osd_scrub_sleep=2 --osd_pool_default_size=3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-dump/0 --osd-journal=td/osd-scrub-dump/0/journal --chdir= --run-dir=td/osd-scrub-dump '--admin-socket=/tmp/ceph-asok.43630/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-dump/$name.log' '--pid-file=td/osd-scrub-dump/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQC5Pv5dlfCaChAAsm1G66on5W6q/99NmRUn5Q== --osd-uuid af8f9195-a7a7-447d-b86e-cc8117fcdf29 2019-12-21T15:48:09.484 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:48:09.482+0000 7f9d883bdec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T15:48:09.486 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:48:09.484+0000 7f9d883bdec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T15:48:09.488 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:48:09.486+0000 7f9d883bdec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T15:48:09.489 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:48:09.487+0000 7f9d883bdec0 -1 bluestore(td/osd-scrub-dump/0/block) _read_bdev_label failed to open td/osd-scrub-dump/0/block: (2) No such file or directory 2019-12-21T15:48:09.489 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:48:09.487+0000 7f9d883bdec0 -1 bluestore(td/osd-scrub-dump/0/block) _read_bdev_label failed to open td/osd-scrub-dump/0/block: (2) No such file or directory 2019-12-21T15:48:09.489 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:48:09.487+0000 7f9d883bdec0 -1 bluestore(td/osd-scrub-dump/0/block) _read_bdev_label failed to open td/osd-scrub-dump/0/block: (2) No such file or directory 2019-12-21T15:48:09.532 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:48:09.530+0000 7f9d883bdec0 -1 bluestore(td/osd-scrub-dump/0) _read_fsid unparsable uuid 2019-12-21T15:48:12.548 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: local key_fn=td/osd-scrub-dump/0/keyring 2019-12-21T15:48:12.549 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: cat 2019-12-21T15:48:12.549 INFO:tasks.workunit.client.0.smithi172.stdout:adding osd0 key to auth repository 2019-12-21T15:48:12.550 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: echo adding osd0 key to auth repository 2019-12-21T15:48:12.550 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph -i td/osd-scrub-dump/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2019-12-21T15:48:12.759 INFO:tasks.workunit.client.0.smithi172.stdout:start osd.0 2019-12-21T15:48:12.760 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo start osd.0 2019-12-21T15:48:12.760 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph-osd -i 0 --fsid=19b44843-6158-4426-874e-bcacde9394a0 --auth-supported=none --mon-host=127.0.0.1:7184 --osd_max_scrubs=4 --osd_scrub_sleep=0 --osd_scrub_chunk_max=5 --osd_scrub_sleep=2 --osd_pool_default_size=3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-dump/0 --osd-journal=td/osd-scrub-dump/0/journal --chdir= --run-dir=td/osd-scrub-dump '--admin-socket=/tmp/ceph-asok.43630/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-dump/$name.log' '--pid-file=td/osd-scrub-dump/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2019-12-21T15:48:12.760 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: jq '.flags_set[]' 2019-12-21T15:48:12.760 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: grep -q '"noup"' 2019-12-21T15:48:12.760 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph osd dump --format=json 2019-12-21T15:48:12.784 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:48:12.781+0000 7f9939235ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T15:48:12.785 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:48:12.783+0000 7f9939235ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T15:48:12.787 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:48:12.784+0000 7f9939235ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T15:48:12.995 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: wait_for_osd up 0 2019-12-21T15:48:12.996 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:914: wait_for_osd: local state=up 2019-12-21T15:48:12.996 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local id=0 2019-12-21T15:48:12.996 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: status=1 2019-12-21T15:48:12.996 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i=0 )) 2019-12-21T15:48:12.996 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i < 300 )) 2019-12-21T15:48:12.996 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: echo 0 2019-12-21T15:48:12.997 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: grep 'osd.0 up' 2019-12-21T15:48:12.997 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: ceph osd dump 2019-12-21T15:48:12.997 INFO:tasks.workunit.client.0.smithi172.stdout:0 2019-12-21T15:48:13.231 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: sleep 1 2019-12-21T15:48:13.305 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:48:13.303+0000 7f9939235ec0 -1 Falling back to public interface 2019-12-21T15:48:14.065 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:48:14.063+0000 7f9939235ec0 -1 osd.0 0 log_to_monitors {default=true} 2019-12-21T15:48:14.232 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i++ )) 2019-12-21T15:48:14.232 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i < 300 )) 2019-12-21T15:48:14.233 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: echo 1 2019-12-21T15:48:14.233 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: ceph osd dump 2019-12-21T15:48:14.233 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: grep 'osd.0 up' 2019-12-21T15:48:14.233 INFO:tasks.workunit.client.0.smithi172.stdout:1 2019-12-21T15:48:14.472 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: sleep 1 2019-12-21T15:48:15.474 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i++ )) 2019-12-21T15:48:15.474 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i < 300 )) 2019-12-21T15:48:15.474 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: echo 2 2019-12-21T15:48:15.474 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: ceph osd dump 2019-12-21T15:48:15.474 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: grep 'osd.0 up' 2019-12-21T15:48:15.475 INFO:tasks.workunit.client.0.smithi172.stdout:2 2019-12-21T15:48:15.722 INFO:tasks.workunit.client.0.smithi172.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/44005,v1:127.0.0.1:6803/44005] [v2:127.0.0.1:6804/44005,v1:127.0.0.1:6805/44005] exists,up af8f9195-a7a7-447d-b86e-cc8117fcdf29 2019-12-21T15:48:15.722 INFO:tasks.workunit.client.0.smithi172.stdout:add osd1 863807f8-4014-401a-8c49-db0c488874f9 2019-12-21T15:48:15.722 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=0 2019-12-21T15:48:15.722 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: break 2019-12-21T15:48:15.722 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: return 0 2019-12-21T15:48:15.723 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:59: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:48:15.723 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:61: TEST_recover_unexpected: run_osd td/osd-scrub-dump 1 2019-12-21T15:48:15.723 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local dir=td/osd-scrub-dump 2019-12-21T15:48:15.723 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: shift 2019-12-21T15:48:15.723 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local id=1 2019-12-21T15:48:15.723 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2019-12-21T15:48:15.724 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local osd_data=td/osd-scrub-dump/1 2019-12-21T15:48:15.724 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local 'ceph_args=--fsid=19b44843-6158-4426-874e-bcacde9394a0 --auth-supported=none --mon-host=127.0.0.1:7184 --osd_max_scrubs=4 --osd_scrub_sleep=0 --osd_scrub_chunk_max=5 --osd_scrub_sleep=2 --osd_pool_default_size=3 ' 2019-12-21T15:48:15.724 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2019-12-21T15:48:15.724 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-journal-size=100' 2019-12-21T15:48:15.724 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2019-12-21T15:48:15.724 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-data=td/osd-scrub-dump/1' 2019-12-21T15:48:15.724 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-dump/1/journal' 2019-12-21T15:48:15.725 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --chdir=' 2019-12-21T15:48:15.725 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+= 2019-12-21T15:48:15.725 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --run-dir=td/osd-scrub-dump' 2019-12-21T15:48:15.725 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: get_asok_path 2019-12-21T15:48:15.725 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2019-12-21T15:48:15.725 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2019-12-21T15:48:15.726 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2019-12-21T15:48:15.726 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:48:15.726 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:48:15.726 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.43630/$cluster-$name.asok' 2019-12-21T15:48:15.726 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.43630/$cluster-$name.asok' 2019-12-21T15:48:15.726 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --debug-osd=20' 2019-12-21T15:48:15.726 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-ms=1' 2019-12-21T15:48:15.727 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-monc=20' 2019-12-21T15:48:15.727 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --log-file=td/osd-scrub-dump/$name.log' 2019-12-21T15:48:15.727 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --pid-file=td/osd-scrub-dump/$name.pid' 2019-12-21T15:48:15.727 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2019-12-21T15:48:15.728 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2019-12-21T15:48:15.728 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2019-12-21T15:48:15.728 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' ' 2019-12-21T15:48:15.728 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+= 2019-12-21T15:48:15.729 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: mkdir -p td/osd-scrub-dump/1 2019-12-21T15:48:15.729 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: uuidgen 2019-12-21T15:48:15.729 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: local uuid=863807f8-4014-401a-8c49-db0c488874f9 2019-12-21T15:48:15.729 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: echo 'add osd1 863807f8-4014-401a-8c49-db0c488874f9' 2019-12-21T15:48:15.730 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: ceph-authtool --gen-print-key 2019-12-21T15:48:15.736 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: OSD_SECRET=AQC/Pv5dQWDPKxAAI5UJOhziK5AOGFkk2xR/hQ== 2019-12-21T15:48:15.737 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo '{"cephx_secret": "AQC/Pv5dQWDPKxAAI5UJOhziK5AOGFkk2xR/hQ=="}' 2019-12-21T15:48:15.737 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph osd new 863807f8-4014-401a-8c49-db0c488874f9 -i td/osd-scrub-dump/1/new.json 2019-12-21T15:48:16.095 INFO:tasks.workunit.client.0.smithi172.stdout:1 2019-12-21T15:48:16.105 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: rm td/osd-scrub-dump/1/new.json 2019-12-21T15:48:16.106 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-osd -i 1 --fsid=19b44843-6158-4426-874e-bcacde9394a0 --auth-supported=none --mon-host=127.0.0.1:7184 --osd_max_scrubs=4 --osd_scrub_sleep=0 --osd_scrub_chunk_max=5 --osd_scrub_sleep=2 --osd_pool_default_size=3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-dump/1 --osd-journal=td/osd-scrub-dump/1/journal --chdir= --run-dir=td/osd-scrub-dump '--admin-socket=/tmp/ceph-asok.43630/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-dump/$name.log' '--pid-file=td/osd-scrub-dump/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQC/Pv5dQWDPKxAAI5UJOhziK5AOGFkk2xR/hQ== --osd-uuid 863807f8-4014-401a-8c49-db0c488874f9 2019-12-21T15:48:16.131 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:48:16.128+0000 7fd4a20feec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T15:48:16.133 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:48:16.130+0000 7fd4a20feec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T15:48:16.135 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:48:16.132+0000 7fd4a20feec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T15:48:16.136 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:48:16.133+0000 7fd4a20feec0 -1 bluestore(td/osd-scrub-dump/1/block) _read_bdev_label failed to open td/osd-scrub-dump/1/block: (2) No such file or directory 2019-12-21T15:48:16.136 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:48:16.133+0000 7fd4a20feec0 -1 bluestore(td/osd-scrub-dump/1/block) _read_bdev_label failed to open td/osd-scrub-dump/1/block: (2) No such file or directory 2019-12-21T15:48:16.136 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:48:16.133+0000 7fd4a20feec0 -1 bluestore(td/osd-scrub-dump/1/block) _read_bdev_label failed to open td/osd-scrub-dump/1/block: (2) No such file or directory 2019-12-21T15:48:16.197 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:48:16.195+0000 7fd4a20feec0 -1 bluestore(td/osd-scrub-dump/1) _read_fsid unparsable uuid 2019-12-21T15:48:19.215 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: local key_fn=td/osd-scrub-dump/1/keyring 2019-12-21T15:48:19.216 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: cat 2019-12-21T15:48:19.217 INFO:tasks.workunit.client.0.smithi172.stdout:adding osd1 key to auth repository 2019-12-21T15:48:19.217 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: echo adding osd1 key to auth repository 2019-12-21T15:48:19.217 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph -i td/osd-scrub-dump/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2019-12-21T15:48:19.506 INFO:tasks.workunit.client.0.smithi172.stdout:start osd.1 2019-12-21T15:48:19.507 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo start osd.1 2019-12-21T15:48:19.507 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph-osd -i 1 --fsid=19b44843-6158-4426-874e-bcacde9394a0 --auth-supported=none --mon-host=127.0.0.1:7184 --osd_max_scrubs=4 --osd_scrub_sleep=0 --osd_scrub_chunk_max=5 --osd_scrub_sleep=2 --osd_pool_default_size=3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-dump/1 --osd-journal=td/osd-scrub-dump/1/journal --chdir= --run-dir=td/osd-scrub-dump '--admin-socket=/tmp/ceph-asok.43630/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-dump/$name.log' '--pid-file=td/osd-scrub-dump/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2019-12-21T15:48:19.507 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: jq '.flags_set[]' 2019-12-21T15:48:19.507 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: grep -q '"noup"' 2019-12-21T15:48:19.507 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph osd dump --format=json 2019-12-21T15:48:19.529 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:48:19.526+0000 7f87357f2ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T15:48:19.531 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:48:19.528+0000 7f87357f2ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T15:48:19.532 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:48:19.529+0000 7f87357f2ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T15:48:19.742 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: wait_for_osd up 1 2019-12-21T15:48:19.742 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:914: wait_for_osd: local state=up 2019-12-21T15:48:19.742 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local id=1 2019-12-21T15:48:19.743 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: status=1 2019-12-21T15:48:19.743 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i=0 )) 2019-12-21T15:48:19.743 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i < 300 )) 2019-12-21T15:48:19.743 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: echo 0 2019-12-21T15:48:19.743 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: ceph osd dump 2019-12-21T15:48:19.743 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: grep 'osd.1 up' 2019-12-21T15:48:19.744 INFO:tasks.workunit.client.0.smithi172.stdout:0 2019-12-21T15:48:19.976 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: sleep 1 2019-12-21T15:48:20.047 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:48:20.044+0000 7f87357f2ec0 -1 Falling back to public interface 2019-12-21T15:48:20.789 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:48:20.787+0000 7f87357f2ec0 -1 osd.1 0 log_to_monitors {default=true} 2019-12-21T15:48:20.978 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i++ )) 2019-12-21T15:48:20.978 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i < 300 )) 2019-12-21T15:48:20.978 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: echo 1 2019-12-21T15:48:20.978 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: ceph osd dump 2019-12-21T15:48:20.978 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: grep 'osd.1 up' 2019-12-21T15:48:20.978 INFO:tasks.workunit.client.0.smithi172.stdout:1 2019-12-21T15:48:21.254 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: sleep 1 2019-12-21T15:48:22.255 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i++ )) 2019-12-21T15:48:22.255 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i < 300 )) 2019-12-21T15:48:22.255 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: echo 2 2019-12-21T15:48:22.255 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: ceph osd dump 2019-12-21T15:48:22.255 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: grep 'osd.1 up' 2019-12-21T15:48:22.256 INFO:tasks.workunit.client.0.smithi172.stdout:2 2019-12-21T15:48:22.496 INFO:tasks.workunit.client.0.smithi172.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/44382,v1:127.0.0.1:6811/44382] [v2:127.0.0.1:6812/44382,v1:127.0.0.1:6813/44382] exists,up 863807f8-4014-401a-8c49-db0c488874f9 2019-12-21T15:48:22.496 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=0 2019-12-21T15:48:22.497 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: break 2019-12-21T15:48:22.497 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: return 0 2019-12-21T15:48:22.497 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:59: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:48:22.497 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:61: TEST_recover_unexpected: run_osd td/osd-scrub-dump 2 2019-12-21T15:48:22.497 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local dir=td/osd-scrub-dump 2019-12-21T15:48:22.497 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: shift 2019-12-21T15:48:22.497 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local id=2 2019-12-21T15:48:22.498 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2019-12-21T15:48:22.498 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local osd_data=td/osd-scrub-dump/2 2019-12-21T15:48:22.498 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local 'ceph_args=--fsid=19b44843-6158-4426-874e-bcacde9394a0 --auth-supported=none --mon-host=127.0.0.1:7184 --osd_max_scrubs=4 --osd_scrub_sleep=0 --osd_scrub_chunk_max=5 --osd_scrub_sleep=2 --osd_pool_default_size=3 ' 2019-12-21T15:48:22.498 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2019-12-21T15:48:22.498 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-journal-size=100' 2019-12-21T15:48:22.498 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2019-12-21T15:48:22.498 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-data=td/osd-scrub-dump/2' 2019-12-21T15:48:22.499 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-dump/2/journal' 2019-12-21T15:48:22.499 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --chdir=' 2019-12-21T15:48:22.499 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+= 2019-12-21T15:48:22.499 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --run-dir=td/osd-scrub-dump' 2019-12-21T15:48:22.502 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: get_asok_path 2019-12-21T15:48:22.502 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2019-12-21T15:48:22.502 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2019-12-21T15:48:22.502 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2019-12-21T15:48:22.502 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:48:22.503 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:48:22.503 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.43630/$cluster-$name.asok' 2019-12-21T15:48:22.503 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.43630/$cluster-$name.asok' 2019-12-21T15:48:22.503 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --debug-osd=20' 2019-12-21T15:48:22.503 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-ms=1' 2019-12-21T15:48:22.503 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-monc=20' 2019-12-21T15:48:22.503 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --log-file=td/osd-scrub-dump/$name.log' 2019-12-21T15:48:22.504 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --pid-file=td/osd-scrub-dump/$name.pid' 2019-12-21T15:48:22.507 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2019-12-21T15:48:22.507 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2019-12-21T15:48:22.507 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2019-12-21T15:48:22.508 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' ' 2019-12-21T15:48:22.508 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+= 2019-12-21T15:48:22.508 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: mkdir -p td/osd-scrub-dump/2 2019-12-21T15:48:22.509 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: uuidgen 2019-12-21T15:48:22.509 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: local uuid=90e9ae31-400f-45c4-8eea-b15d5da90374 2019-12-21T15:48:22.509 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: echo 'add osd2 90e9ae31-400f-45c4-8eea-b15d5da90374' 2019-12-21T15:48:22.509 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: ceph-authtool --gen-print-key 2019-12-21T15:48:22.509 INFO:tasks.workunit.client.0.smithi172.stdout:add osd2 90e9ae31-400f-45c4-8eea-b15d5da90374 2019-12-21T15:48:22.513 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: OSD_SECRET=AQDGPv5d+j9+HhAAtVMEh+6eXzz34FxAD/OXdQ== 2019-12-21T15:48:22.513 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo '{"cephx_secret": "AQDGPv5d+j9+HhAAtVMEh+6eXzz34FxAD/OXdQ=="}' 2019-12-21T15:48:22.513 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph osd new 90e9ae31-400f-45c4-8eea-b15d5da90374 -i td/osd-scrub-dump/2/new.json 2019-12-21T15:48:22.831 INFO:tasks.workunit.client.0.smithi172.stdout:2 2019-12-21T15:48:22.840 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: rm td/osd-scrub-dump/2/new.json 2019-12-21T15:48:22.840 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-osd -i 2 --fsid=19b44843-6158-4426-874e-bcacde9394a0 --auth-supported=none --mon-host=127.0.0.1:7184 --osd_max_scrubs=4 --osd_scrub_sleep=0 --osd_scrub_chunk_max=5 --osd_scrub_sleep=2 --osd_pool_default_size=3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-dump/2 --osd-journal=td/osd-scrub-dump/2/journal --chdir= --run-dir=td/osd-scrub-dump '--admin-socket=/tmp/ceph-asok.43630/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-dump/$name.log' '--pid-file=td/osd-scrub-dump/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDGPv5d+j9+HhAAtVMEh+6eXzz34FxAD/OXdQ== --osd-uuid 90e9ae31-400f-45c4-8eea-b15d5da90374 2019-12-21T15:48:22.866 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:48:22.863+0000 7fc4a82a1ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T15:48:22.868 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:48:22.865+0000 7fc4a82a1ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T15:48:22.870 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:48:22.867+0000 7fc4a82a1ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T15:48:22.870 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:48:22.867+0000 7fc4a82a1ec0 -1 bluestore(td/osd-scrub-dump/2/block) _read_bdev_label failed to open td/osd-scrub-dump/2/block: (2) No such file or directory 2019-12-21T15:48:22.871 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:48:22.867+0000 7fc4a82a1ec0 -1 bluestore(td/osd-scrub-dump/2/block) _read_bdev_label failed to open td/osd-scrub-dump/2/block: (2) No such file or directory 2019-12-21T15:48:22.871 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:48:22.867+0000 7fc4a82a1ec0 -1 bluestore(td/osd-scrub-dump/2/block) _read_bdev_label failed to open td/osd-scrub-dump/2/block: (2) No such file or directory 2019-12-21T15:48:22.930 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:48:22.927+0000 7fc4a82a1ec0 -1 bluestore(td/osd-scrub-dump/2) _read_fsid unparsable uuid 2019-12-21T15:48:25.916 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: local key_fn=td/osd-scrub-dump/2/keyring 2019-12-21T15:48:25.916 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: cat 2019-12-21T15:48:25.917 INFO:tasks.workunit.client.0.smithi172.stdout:adding osd2 key to auth repository 2019-12-21T15:48:25.917 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: echo adding osd2 key to auth repository 2019-12-21T15:48:25.917 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph -i td/osd-scrub-dump/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2019-12-21T15:48:26.207 INFO:tasks.workunit.client.0.smithi172.stdout:start osd.2 2019-12-21T15:48:26.207 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo start osd.2 2019-12-21T15:48:26.208 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph-osd -i 2 --fsid=19b44843-6158-4426-874e-bcacde9394a0 --auth-supported=none --mon-host=127.0.0.1:7184 --osd_max_scrubs=4 --osd_scrub_sleep=0 --osd_scrub_chunk_max=5 --osd_scrub_sleep=2 --osd_pool_default_size=3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-dump/2 --osd-journal=td/osd-scrub-dump/2/journal --chdir= --run-dir=td/osd-scrub-dump '--admin-socket=/tmp/ceph-asok.43630/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-dump/$name.log' '--pid-file=td/osd-scrub-dump/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2019-12-21T15:48:26.208 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph osd dump --format=json 2019-12-21T15:48:26.208 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: jq '.flags_set[]' 2019-12-21T15:48:26.208 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: grep -q '"noup"' 2019-12-21T15:48:26.231 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:48:26.228+0000 7f69ea99cec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T15:48:26.232 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:48:26.230+0000 7f69ea99cec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T15:48:26.233 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:48:26.231+0000 7f69ea99cec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T15:48:26.443 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: wait_for_osd up 2 2019-12-21T15:48:26.443 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:914: wait_for_osd: local state=up 2019-12-21T15:48:26.443 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local id=2 2019-12-21T15:48:26.444 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: status=1 2019-12-21T15:48:26.444 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i=0 )) 2019-12-21T15:48:26.444 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i < 300 )) 2019-12-21T15:48:26.444 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: echo 0 2019-12-21T15:48:26.444 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: grep 'osd.2 up' 2019-12-21T15:48:26.444 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: ceph osd dump 2019-12-21T15:48:26.445 INFO:tasks.workunit.client.0.smithi172.stdout:0 2019-12-21T15:48:26.678 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: sleep 1 2019-12-21T15:48:26.749 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:48:26.747+0000 7f69ea99cec0 -1 Falling back to public interface 2019-12-21T15:48:27.512 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:48:27.509+0000 7f69ea99cec0 -1 osd.2 0 log_to_monitors {default=true} 2019-12-21T15:48:27.680 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i++ )) 2019-12-21T15:48:27.680 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i < 300 )) 2019-12-21T15:48:27.680 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: echo 1 2019-12-21T15:48:27.680 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: grep 'osd.2 up' 2019-12-21T15:48:27.680 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: ceph osd dump 2019-12-21T15:48:27.681 INFO:tasks.workunit.client.0.smithi172.stdout:1 2019-12-21T15:48:27.941 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: sleep 1 2019-12-21T15:48:28.943 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i++ )) 2019-12-21T15:48:28.943 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i < 300 )) 2019-12-21T15:48:28.943 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: echo 2 2019-12-21T15:48:28.943 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: ceph osd dump 2019-12-21T15:48:28.943 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: grep 'osd.2 up' 2019-12-21T15:48:28.944 INFO:tasks.workunit.client.0.smithi172.stdout:2 2019-12-21T15:48:29.186 INFO:tasks.workunit.client.0.smithi172.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/44759,v1:127.0.0.1:6819/44759] [v2:127.0.0.1:6820/44759,v1:127.0.0.1:6821/44759] exists,up 90e9ae31-400f-45c4-8eea-b15d5da90374 2019-12-21T15:48:29.186 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=0 2019-12-21T15:48:29.187 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: break 2019-12-21T15:48:29.187 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: return 0 2019-12-21T15:48:29.187 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:59: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:48:29.187 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:61: TEST_recover_unexpected: run_osd td/osd-scrub-dump 3 2019-12-21T15:48:29.187 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local dir=td/osd-scrub-dump 2019-12-21T15:48:29.187 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: shift 2019-12-21T15:48:29.187 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local id=3 2019-12-21T15:48:29.188 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2019-12-21T15:48:29.188 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local osd_data=td/osd-scrub-dump/3 2019-12-21T15:48:29.188 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local 'ceph_args=--fsid=19b44843-6158-4426-874e-bcacde9394a0 --auth-supported=none --mon-host=127.0.0.1:7184 --osd_max_scrubs=4 --osd_scrub_sleep=0 --osd_scrub_chunk_max=5 --osd_scrub_sleep=2 --osd_pool_default_size=3 ' 2019-12-21T15:48:29.188 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2019-12-21T15:48:29.188 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-journal-size=100' 2019-12-21T15:48:29.188 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2019-12-21T15:48:29.188 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-data=td/osd-scrub-dump/3' 2019-12-21T15:48:29.189 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-dump/3/journal' 2019-12-21T15:48:29.189 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --chdir=' 2019-12-21T15:48:29.189 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+= 2019-12-21T15:48:29.189 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --run-dir=td/osd-scrub-dump' 2019-12-21T15:48:29.189 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: get_asok_path 2019-12-21T15:48:29.189 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2019-12-21T15:48:29.189 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2019-12-21T15:48:29.189 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2019-12-21T15:48:29.190 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:48:29.190 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:48:29.190 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.43630/$cluster-$name.asok' 2019-12-21T15:48:29.190 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.43630/$cluster-$name.asok' 2019-12-21T15:48:29.190 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --debug-osd=20' 2019-12-21T15:48:29.190 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-ms=1' 2019-12-21T15:48:29.190 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-monc=20' 2019-12-21T15:48:29.191 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --log-file=td/osd-scrub-dump/$name.log' 2019-12-21T15:48:29.191 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --pid-file=td/osd-scrub-dump/$name.pid' 2019-12-21T15:48:29.191 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2019-12-21T15:48:29.191 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2019-12-21T15:48:29.191 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2019-12-21T15:48:29.191 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' ' 2019-12-21T15:48:29.191 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+= 2019-12-21T15:48:29.191 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: mkdir -p td/osd-scrub-dump/3 2019-12-21T15:48:29.192 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: uuidgen 2019-12-21T15:48:29.192 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: local uuid=5b0141e7-8c53-4724-a0e4-1ecbd58e7067 2019-12-21T15:48:29.193 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: echo 'add osd3 5b0141e7-8c53-4724-a0e4-1ecbd58e7067' 2019-12-21T15:48:29.193 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: ceph-authtool --gen-print-key 2019-12-21T15:48:29.193 INFO:tasks.workunit.client.0.smithi172.stdout:add osd3 5b0141e7-8c53-4724-a0e4-1ecbd58e7067 2019-12-21T15:48:29.204 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: OSD_SECRET=AQDNPv5d3soQDBAAnEe6gb1XlqC7ayUaxFGmhA== 2019-12-21T15:48:29.204 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo '{"cephx_secret": "AQDNPv5d3soQDBAAnEe6gb1XlqC7ayUaxFGmhA=="}' 2019-12-21T15:48:29.204 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph osd new 5b0141e7-8c53-4724-a0e4-1ecbd58e7067 -i td/osd-scrub-dump/3/new.json 2019-12-21T15:48:29.503 INFO:tasks.workunit.client.0.smithi172.stdout:3 2019-12-21T15:48:29.513 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: rm td/osd-scrub-dump/3/new.json 2019-12-21T15:48:29.514 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-osd -i 3 --fsid=19b44843-6158-4426-874e-bcacde9394a0 --auth-supported=none --mon-host=127.0.0.1:7184 --osd_max_scrubs=4 --osd_scrub_sleep=0 --osd_scrub_chunk_max=5 --osd_scrub_sleep=2 --osd_pool_default_size=3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-dump/3 --osd-journal=td/osd-scrub-dump/3/journal --chdir= --run-dir=td/osd-scrub-dump '--admin-socket=/tmp/ceph-asok.43630/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-dump/$name.log' '--pid-file=td/osd-scrub-dump/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDNPv5d3soQDBAAnEe6gb1XlqC7ayUaxFGmhA== --osd-uuid 5b0141e7-8c53-4724-a0e4-1ecbd58e7067 2019-12-21T15:48:29.539 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:48:29.536+0000 7f3f68fadec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T15:48:29.541 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:48:29.538+0000 7f3f68fadec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T15:48:29.542 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:48:29.540+0000 7f3f68fadec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T15:48:29.543 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:48:29.541+0000 7f3f68fadec0 -1 bluestore(td/osd-scrub-dump/3/block) _read_bdev_label failed to open td/osd-scrub-dump/3/block: (2) No such file or directory 2019-12-21T15:48:29.543 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:48:29.541+0000 7f3f68fadec0 -1 bluestore(td/osd-scrub-dump/3/block) _read_bdev_label failed to open td/osd-scrub-dump/3/block: (2) No such file or directory 2019-12-21T15:48:29.543 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:48:29.541+0000 7f3f68fadec0 -1 bluestore(td/osd-scrub-dump/3/block) _read_bdev_label failed to open td/osd-scrub-dump/3/block: (2) No such file or directory 2019-12-21T15:48:29.586 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:48:29.584+0000 7f3f68fadec0 -1 bluestore(td/osd-scrub-dump/3) _read_fsid unparsable uuid 2019-12-21T15:48:32.540 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: local key_fn=td/osd-scrub-dump/3/keyring 2019-12-21T15:48:32.540 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: cat 2019-12-21T15:48:32.541 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: echo adding osd3 key to auth repository 2019-12-21T15:48:32.542 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph -i td/osd-scrub-dump/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2019-12-21T15:48:32.542 INFO:tasks.workunit.client.0.smithi172.stdout:adding osd3 key to auth repository 2019-12-21T15:48:32.835 INFO:tasks.workunit.client.0.smithi172.stdout:start osd.3 2019-12-21T15:48:32.835 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo start osd.3 2019-12-21T15:48:32.835 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph-osd -i 3 --fsid=19b44843-6158-4426-874e-bcacde9394a0 --auth-supported=none --mon-host=127.0.0.1:7184 --osd_max_scrubs=4 --osd_scrub_sleep=0 --osd_scrub_chunk_max=5 --osd_scrub_sleep=2 --osd_pool_default_size=3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-dump/3 --osd-journal=td/osd-scrub-dump/3/journal --chdir= --run-dir=td/osd-scrub-dump '--admin-socket=/tmp/ceph-asok.43630/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-dump/$name.log' '--pid-file=td/osd-scrub-dump/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2019-12-21T15:48:32.835 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: jq '.flags_set[]' 2019-12-21T15:48:32.835 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: grep -q '"noup"' 2019-12-21T15:48:32.836 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph osd dump --format=json 2019-12-21T15:48:32.858 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:48:32.857+0000 7f6a20c1dec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T15:48:32.860 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:48:32.858+0000 7f6a20c1dec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T15:48:32.861 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:48:32.860+0000 7f6a20c1dec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T15:48:33.071 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: wait_for_osd up 3 2019-12-21T15:48:33.072 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:914: wait_for_osd: local state=up 2019-12-21T15:48:33.072 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local id=3 2019-12-21T15:48:33.072 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: status=1 2019-12-21T15:48:33.072 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i=0 )) 2019-12-21T15:48:33.072 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i < 300 )) 2019-12-21T15:48:33.072 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: echo 0 2019-12-21T15:48:33.072 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: ceph osd dump 2019-12-21T15:48:33.073 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: grep 'osd.3 up' 2019-12-21T15:48:33.073 INFO:tasks.workunit.client.0.smithi172.stdout:0 2019-12-21T15:48:33.305 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: sleep 1 2019-12-21T15:48:33.377 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:48:33.375+0000 7f6a20c1dec0 -1 Falling back to public interface 2019-12-21T15:48:34.111 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:48:34.109+0000 7f6a20c1dec0 -1 osd.3 0 log_to_monitors {default=true} 2019-12-21T15:48:34.307 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i++ )) 2019-12-21T15:48:34.307 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i < 300 )) 2019-12-21T15:48:34.307 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: echo 1 2019-12-21T15:48:34.308 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: grep 'osd.3 up' 2019-12-21T15:48:34.308 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: ceph osd dump 2019-12-21T15:48:34.308 INFO:tasks.workunit.client.0.smithi172.stdout:1 2019-12-21T15:48:34.634 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: sleep 1 2019-12-21T15:48:35.636 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i++ )) 2019-12-21T15:48:35.636 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i < 300 )) 2019-12-21T15:48:35.636 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: echo 2 2019-12-21T15:48:35.637 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: ceph osd dump 2019-12-21T15:48:35.637 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: grep 'osd.3 up' 2019-12-21T15:48:35.637 INFO:tasks.workunit.client.0.smithi172.stdout:2 2019-12-21T15:48:35.877 INFO:tasks.workunit.client.0.smithi172.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/45136,v1:127.0.0.1:6827/45136] [v2:127.0.0.1:6828/45136,v1:127.0.0.1:6829/45136] exists,up 5b0141e7-8c53-4724-a0e4-1ecbd58e7067 2019-12-21T15:48:35.877 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=0 2019-12-21T15:48:35.877 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: break 2019-12-21T15:48:35.877 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: return 0 2019-12-21T15:48:35.878 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:59: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:48:35.878 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:61: TEST_recover_unexpected: run_osd td/osd-scrub-dump 4 2019-12-21T15:48:35.878 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local dir=td/osd-scrub-dump 2019-12-21T15:48:35.878 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: shift 2019-12-21T15:48:35.878 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local id=4 2019-12-21T15:48:35.878 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2019-12-21T15:48:35.878 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local osd_data=td/osd-scrub-dump/4 2019-12-21T15:48:35.879 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local 'ceph_args=--fsid=19b44843-6158-4426-874e-bcacde9394a0 --auth-supported=none --mon-host=127.0.0.1:7184 --osd_max_scrubs=4 --osd_scrub_sleep=0 --osd_scrub_chunk_max=5 --osd_scrub_sleep=2 --osd_pool_default_size=3 ' 2019-12-21T15:48:35.879 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2019-12-21T15:48:35.879 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-journal-size=100' 2019-12-21T15:48:35.879 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2019-12-21T15:48:35.879 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-data=td/osd-scrub-dump/4' 2019-12-21T15:48:35.879 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-dump/4/journal' 2019-12-21T15:48:35.879 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --chdir=' 2019-12-21T15:48:35.880 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+= 2019-12-21T15:48:35.880 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --run-dir=td/osd-scrub-dump' 2019-12-21T15:48:35.880 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: get_asok_path 2019-12-21T15:48:35.880 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2019-12-21T15:48:35.880 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2019-12-21T15:48:35.880 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2019-12-21T15:48:35.880 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:48:35.881 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:48:35.881 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.43630/$cluster-$name.asok' 2019-12-21T15:48:35.882 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.43630/$cluster-$name.asok' 2019-12-21T15:48:35.882 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --debug-osd=20' 2019-12-21T15:48:35.882 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-ms=1' 2019-12-21T15:48:35.882 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-monc=20' 2019-12-21T15:48:35.882 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --log-file=td/osd-scrub-dump/$name.log' 2019-12-21T15:48:35.882 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --pid-file=td/osd-scrub-dump/$name.pid' 2019-12-21T15:48:35.882 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2019-12-21T15:48:35.883 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2019-12-21T15:48:35.883 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2019-12-21T15:48:35.883 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' ' 2019-12-21T15:48:35.883 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+= 2019-12-21T15:48:35.883 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: mkdir -p td/osd-scrub-dump/4 2019-12-21T15:48:35.884 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: uuidgen 2019-12-21T15:48:35.884 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: local uuid=098cfcfe-9f18-4ee3-a90e-89cff1f51d88 2019-12-21T15:48:35.884 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: echo 'add osd4 098cfcfe-9f18-4ee3-a90e-89cff1f51d88' 2019-12-21T15:48:35.885 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: ceph-authtool --gen-print-key 2019-12-21T15:48:35.885 INFO:tasks.workunit.client.0.smithi172.stdout:add osd4 098cfcfe-9f18-4ee3-a90e-89cff1f51d88 2019-12-21T15:48:35.894 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: OSD_SECRET=AQDTPv5dtRAvNRAAXLu94K3/nc/X+UPiKIPWtg== 2019-12-21T15:48:35.894 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo '{"cephx_secret": "AQDTPv5dtRAvNRAAXLu94K3/nc/X+UPiKIPWtg=="}' 2019-12-21T15:48:35.894 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph osd new 098cfcfe-9f18-4ee3-a90e-89cff1f51d88 -i td/osd-scrub-dump/4/new.json 2019-12-21T15:48:36.211 INFO:tasks.workunit.client.0.smithi172.stdout:4 2019-12-21T15:48:36.220 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: rm td/osd-scrub-dump/4/new.json 2019-12-21T15:48:36.221 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-osd -i 4 --fsid=19b44843-6158-4426-874e-bcacde9394a0 --auth-supported=none --mon-host=127.0.0.1:7184 --osd_max_scrubs=4 --osd_scrub_sleep=0 --osd_scrub_chunk_max=5 --osd_scrub_sleep=2 --osd_pool_default_size=3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-dump/4 --osd-journal=td/osd-scrub-dump/4/journal --chdir= --run-dir=td/osd-scrub-dump '--admin-socket=/tmp/ceph-asok.43630/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-dump/$name.log' '--pid-file=td/osd-scrub-dump/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDTPv5dtRAvNRAAXLu94K3/nc/X+UPiKIPWtg== --osd-uuid 098cfcfe-9f18-4ee3-a90e-89cff1f51d88 2019-12-21T15:48:36.247 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:48:36.245+0000 7f9314eb5ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T15:48:36.252 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:48:36.250+0000 7f9314eb5ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T15:48:36.253 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:48:36.252+0000 7f9314eb5ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T15:48:36.254 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:48:36.252+0000 7f9314eb5ec0 -1 bluestore(td/osd-scrub-dump/4/block) _read_bdev_label failed to open td/osd-scrub-dump/4/block: (2) No such file or directory 2019-12-21T15:48:36.254 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:48:36.252+0000 7f9314eb5ec0 -1 bluestore(td/osd-scrub-dump/4/block) _read_bdev_label failed to open td/osd-scrub-dump/4/block: (2) No such file or directory 2019-12-21T15:48:36.254 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:48:36.252+0000 7f9314eb5ec0 -1 bluestore(td/osd-scrub-dump/4/block) _read_bdev_label failed to open td/osd-scrub-dump/4/block: (2) No such file or directory 2019-12-21T15:48:36.310 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:48:36.308+0000 7f9314eb5ec0 -1 bluestore(td/osd-scrub-dump/4) _read_fsid unparsable uuid 2019-12-21T15:48:39.436 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: local key_fn=td/osd-scrub-dump/4/keyring 2019-12-21T15:48:39.436 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: cat 2019-12-21T15:48:39.437 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: echo adding osd4 key to auth repository 2019-12-21T15:48:39.438 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph -i td/osd-scrub-dump/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2019-12-21T15:48:39.438 INFO:tasks.workunit.client.0.smithi172.stdout:adding osd4 key to auth repository 2019-12-21T15:48:39.728 INFO:tasks.workunit.client.0.smithi172.stdout:start osd.4 2019-12-21T15:48:39.728 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo start osd.4 2019-12-21T15:48:39.728 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph-osd -i 4 --fsid=19b44843-6158-4426-874e-bcacde9394a0 --auth-supported=none --mon-host=127.0.0.1:7184 --osd_max_scrubs=4 --osd_scrub_sleep=0 --osd_scrub_chunk_max=5 --osd_scrub_sleep=2 --osd_pool_default_size=3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-dump/4 --osd-journal=td/osd-scrub-dump/4/journal --chdir= --run-dir=td/osd-scrub-dump '--admin-socket=/tmp/ceph-asok.43630/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-dump/$name.log' '--pid-file=td/osd-scrub-dump/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2019-12-21T15:48:39.728 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph osd dump --format=json 2019-12-21T15:48:39.728 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: jq '.flags_set[]' 2019-12-21T15:48:39.729 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: grep -q '"noup"' 2019-12-21T15:48:39.751 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:48:39.750+0000 7f991523eec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T15:48:39.753 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:48:39.751+0000 7f991523eec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T15:48:39.754 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:48:39.753+0000 7f991523eec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T15:48:39.964 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: wait_for_osd up 4 2019-12-21T15:48:39.964 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:914: wait_for_osd: local state=up 2019-12-21T15:48:39.964 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local id=4 2019-12-21T15:48:39.965 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: status=1 2019-12-21T15:48:39.965 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i=0 )) 2019-12-21T15:48:39.965 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i < 300 )) 2019-12-21T15:48:39.965 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: echo 0 2019-12-21T15:48:39.965 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: ceph osd dump 2019-12-21T15:48:39.965 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: grep 'osd.4 up' 2019-12-21T15:48:39.966 INFO:tasks.workunit.client.0.smithi172.stdout:0 2019-12-21T15:48:40.200 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: sleep 1 2019-12-21T15:48:40.271 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:48:40.270+0000 7f991523eec0 -1 Falling back to public interface 2019-12-21T15:48:40.978 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:48:40.977+0000 7f991523eec0 -1 osd.4 0 log_to_monitors {default=true} 2019-12-21T15:48:41.201 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i++ )) 2019-12-21T15:48:41.201 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i < 300 )) 2019-12-21T15:48:41.202 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: echo 1 2019-12-21T15:48:41.202 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: ceph osd dump 2019-12-21T15:48:41.202 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: grep 'osd.4 up' 2019-12-21T15:48:41.202 INFO:tasks.workunit.client.0.smithi172.stdout:1 2019-12-21T15:48:41.455 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: sleep 1 2019-12-21T15:48:42.457 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i++ )) 2019-12-21T15:48:42.457 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i < 300 )) 2019-12-21T15:48:42.457 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: echo 2 2019-12-21T15:48:42.457 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: ceph osd dump 2019-12-21T15:48:42.457 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: grep 'osd.4 up' 2019-12-21T15:48:42.458 INFO:tasks.workunit.client.0.smithi172.stdout:2 2019-12-21T15:48:42.726 INFO:tasks.workunit.client.0.smithi172.stdout:osd.4 up in weight 1 up_from 25 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/45513,v1:127.0.0.1:6835/45513] [v2:127.0.0.1:6836/45513,v1:127.0.0.1:6837/45513] exists,up 098cfcfe-9f18-4ee3-a90e-89cff1f51d88 2019-12-21T15:48:42.727 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=0 2019-12-21T15:48:42.727 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: break 2019-12-21T15:48:42.727 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: return 0 2019-12-21T15:48:42.727 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:59: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:48:42.727 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:61: TEST_recover_unexpected: run_osd td/osd-scrub-dump 5 2019-12-21T15:48:42.727 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local dir=td/osd-scrub-dump 2019-12-21T15:48:42.728 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: shift 2019-12-21T15:48:42.728 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local id=5 2019-12-21T15:48:42.728 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2019-12-21T15:48:42.728 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local osd_data=td/osd-scrub-dump/5 2019-12-21T15:48:42.728 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local 'ceph_args=--fsid=19b44843-6158-4426-874e-bcacde9394a0 --auth-supported=none --mon-host=127.0.0.1:7184 --osd_max_scrubs=4 --osd_scrub_sleep=0 --osd_scrub_chunk_max=5 --osd_scrub_sleep=2 --osd_pool_default_size=3 ' 2019-12-21T15:48:42.728 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2019-12-21T15:48:42.729 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-journal-size=100' 2019-12-21T15:48:42.729 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2019-12-21T15:48:42.729 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-data=td/osd-scrub-dump/5' 2019-12-21T15:48:42.729 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-dump/5/journal' 2019-12-21T15:48:42.729 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --chdir=' 2019-12-21T15:48:42.729 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+= 2019-12-21T15:48:42.730 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --run-dir=td/osd-scrub-dump' 2019-12-21T15:48:42.730 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: get_asok_path 2019-12-21T15:48:42.730 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2019-12-21T15:48:42.731 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2019-12-21T15:48:42.731 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2019-12-21T15:48:42.731 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:48:42.731 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:48:42.731 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.43630/$cluster-$name.asok' 2019-12-21T15:48:42.731 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.43630/$cluster-$name.asok' 2019-12-21T15:48:42.731 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --debug-osd=20' 2019-12-21T15:48:42.732 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-ms=1' 2019-12-21T15:48:42.732 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-monc=20' 2019-12-21T15:48:42.732 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --log-file=td/osd-scrub-dump/$name.log' 2019-12-21T15:48:42.732 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --pid-file=td/osd-scrub-dump/$name.pid' 2019-12-21T15:48:42.732 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2019-12-21T15:48:42.732 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2019-12-21T15:48:42.732 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2019-12-21T15:48:42.733 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' ' 2019-12-21T15:48:42.733 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+= 2019-12-21T15:48:42.734 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: mkdir -p td/osd-scrub-dump/5 2019-12-21T15:48:42.735 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: uuidgen 2019-12-21T15:48:42.735 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: local uuid=03b94da8-338b-42f7-800a-4cf5fab5cdbc 2019-12-21T15:48:42.735 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: echo 'add osd5 03b94da8-338b-42f7-800a-4cf5fab5cdbc' 2019-12-21T15:48:42.735 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: ceph-authtool --gen-print-key 2019-12-21T15:48:42.736 INFO:tasks.workunit.client.0.smithi172.stdout:add osd5 03b94da8-338b-42f7-800a-4cf5fab5cdbc 2019-12-21T15:48:42.742 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: OSD_SECRET=AQDaPv5doJIsLBAAH8GMs1BKDnKer/22zMpAJg== 2019-12-21T15:48:42.743 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo '{"cephx_secret": "AQDaPv5doJIsLBAAH8GMs1BKDnKer/22zMpAJg=="}' 2019-12-21T15:48:42.743 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph osd new 03b94da8-338b-42f7-800a-4cf5fab5cdbc -i td/osd-scrub-dump/5/new.json 2019-12-21T15:48:43.063 INFO:tasks.workunit.client.0.smithi172.stdout:5 2019-12-21T15:48:43.071 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: rm td/osd-scrub-dump/5/new.json 2019-12-21T15:48:43.072 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-osd -i 5 --fsid=19b44843-6158-4426-874e-bcacde9394a0 --auth-supported=none --mon-host=127.0.0.1:7184 --osd_max_scrubs=4 --osd_scrub_sleep=0 --osd_scrub_chunk_max=5 --osd_scrub_sleep=2 --osd_pool_default_size=3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-dump/5 --osd-journal=td/osd-scrub-dump/5/journal --chdir= --run-dir=td/osd-scrub-dump '--admin-socket=/tmp/ceph-asok.43630/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-dump/$name.log' '--pid-file=td/osd-scrub-dump/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDaPv5doJIsLBAAH8GMs1BKDnKer/22zMpAJg== --osd-uuid 03b94da8-338b-42f7-800a-4cf5fab5cdbc 2019-12-21T15:48:43.097 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:48:43.095+0000 7fc324d81ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T15:48:43.098 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:48:43.097+0000 7fc324d81ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T15:48:43.100 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:48:43.099+0000 7fc324d81ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T15:48:43.101 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:48:43.099+0000 7fc324d81ec0 -1 bluestore(td/osd-scrub-dump/5/block) _read_bdev_label failed to open td/osd-scrub-dump/5/block: (2) No such file or directory 2019-12-21T15:48:43.101 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:48:43.099+0000 7fc324d81ec0 -1 bluestore(td/osd-scrub-dump/5/block) _read_bdev_label failed to open td/osd-scrub-dump/5/block: (2) No such file or directory 2019-12-21T15:48:43.101 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:48:43.099+0000 7fc324d81ec0 -1 bluestore(td/osd-scrub-dump/5/block) _read_bdev_label failed to open td/osd-scrub-dump/5/block: (2) No such file or directory 2019-12-21T15:48:43.161 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:48:43.159+0000 7fc324d81ec0 -1 bluestore(td/osd-scrub-dump/5) _read_fsid unparsable uuid 2019-12-21T15:48:46.039 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: local key_fn=td/osd-scrub-dump/5/keyring 2019-12-21T15:48:46.040 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: cat 2019-12-21T15:48:46.041 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: echo adding osd5 key to auth repository 2019-12-21T15:48:46.041 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph -i td/osd-scrub-dump/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2019-12-21T15:48:46.041 INFO:tasks.workunit.client.0.smithi172.stdout:adding osd5 key to auth repository 2019-12-21T15:48:46.332 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo start osd.5 2019-12-21T15:48:46.333 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph osd dump --format=json 2019-12-21T15:48:46.333 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: jq '.flags_set[]' 2019-12-21T15:48:46.333 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: grep -q '"noup"' 2019-12-21T15:48:46.333 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph-osd -i 5 --fsid=19b44843-6158-4426-874e-bcacde9394a0 --auth-supported=none --mon-host=127.0.0.1:7184 --osd_max_scrubs=4 --osd_scrub_sleep=0 --osd_scrub_chunk_max=5 --osd_scrub_sleep=2 --osd_pool_default_size=3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-dump/5 --osd-journal=td/osd-scrub-dump/5/journal --chdir= --run-dir=td/osd-scrub-dump '--admin-socket=/tmp/ceph-asok.43630/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-dump/$name.log' '--pid-file=td/osd-scrub-dump/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2019-12-21T15:48:46.334 INFO:tasks.workunit.client.0.smithi172.stdout:start osd.5 2019-12-21T15:48:46.356 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:48:46.354+0000 7fbc6fbe2ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T15:48:46.357 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:48:46.356+0000 7fbc6fbe2ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T15:48:46.360 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:48:46.357+0000 7fbc6fbe2ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T15:48:46.569 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: wait_for_osd up 5 2019-12-21T15:48:46.569 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:914: wait_for_osd: local state=up 2019-12-21T15:48:46.570 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local id=5 2019-12-21T15:48:46.570 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: status=1 2019-12-21T15:48:46.570 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i=0 )) 2019-12-21T15:48:46.570 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i < 300 )) 2019-12-21T15:48:46.570 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: echo 0 2019-12-21T15:48:46.570 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: ceph osd dump 2019-12-21T15:48:46.570 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: grep 'osd.5 up' 2019-12-21T15:48:46.571 INFO:tasks.workunit.client.0.smithi172.stdout:0 2019-12-21T15:48:46.806 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: sleep 1 2019-12-21T15:48:46.876 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:48:46.874+0000 7fbc6fbe2ec0 -1 Falling back to public interface 2019-12-21T15:48:47.596 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:48:47.594+0000 7fbc6fbe2ec0 -1 osd.5 0 log_to_monitors {default=true} 2019-12-21T15:48:47.807 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i++ )) 2019-12-21T15:48:47.808 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i < 300 )) 2019-12-21T15:48:47.808 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: echo 1 2019-12-21T15:48:47.808 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: ceph osd dump 2019-12-21T15:48:47.808 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: grep 'osd.5 up' 2019-12-21T15:48:47.809 INFO:tasks.workunit.client.0.smithi172.stdout:1 2019-12-21T15:48:48.049 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: sleep 1 2019-12-21T15:48:49.050 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i++ )) 2019-12-21T15:48:49.051 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i < 300 )) 2019-12-21T15:48:49.051 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: echo 2 2019-12-21T15:48:49.051 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: grep 'osd.5 up' 2019-12-21T15:48:49.051 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: ceph osd dump 2019-12-21T15:48:49.051 INFO:tasks.workunit.client.0.smithi172.stdout:2 2019-12-21T15:48:49.291 INFO:tasks.workunit.client.0.smithi172.stdout:osd.5 up in weight 1 up_from 30 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/45890,v1:127.0.0.1:6843/45890] [v2:127.0.0.1:6844/45890,v1:127.0.0.1:6845/45890] exists,up 03b94da8-338b-42f7-800a-4cf5fab5cdbc 2019-12-21T15:48:49.291 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=0 2019-12-21T15:48:49.291 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: break 2019-12-21T15:48:49.291 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: return 0 2019-12-21T15:48:49.292 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:64: TEST_recover_unexpected: seq 1 3 2019-12-21T15:48:49.293 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:64: TEST_recover_unexpected: for i in $(seq 1 $POOLS) 2019-12-21T15:48:49.293 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:66: TEST_recover_unexpected: create_pool test1 16 16 2019-12-21T15:48:49.293 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:539: create_pool: ceph osd pool create test1 16 16 2019-12-21T15:48:49.684 INFO:tasks.workunit.client.0.smithi172.stderr:pool 'test1' created 2019-12-21T15:48:49.691 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: sleep 1 2019-12-21T15:48:50.693 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:64: TEST_recover_unexpected: for i in $(seq 1 $POOLS) 2019-12-21T15:48:50.693 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:66: TEST_recover_unexpected: create_pool test2 16 16 2019-12-21T15:48:50.693 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:539: create_pool: ceph osd pool create test2 16 16 2019-12-21T15:48:51.106 INFO:tasks.workunit.client.0.smithi172.stderr:pool 'test2' created 2019-12-21T15:48:51.115 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: sleep 1 2019-12-21T15:48:52.116 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:64: TEST_recover_unexpected: for i in $(seq 1 $POOLS) 2019-12-21T15:48:52.116 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:66: TEST_recover_unexpected: create_pool test3 16 16 2019-12-21T15:48:52.116 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:539: create_pool: ceph osd pool create test3 16 16 2019-12-21T15:48:52.572 INFO:tasks.workunit.client.0.smithi172.stderr:pool 'test3' created 2019-12-21T15:48:52.579 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: sleep 1 2019-12-21T15:48:53.581 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:69: TEST_recover_unexpected: wait_for_clean 2019-12-21T15:48:53.581 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1548: wait_for_clean: local cmd= 2019-12-21T15:48:53.581 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1549: wait_for_clean: local num_active_clean=-1 2019-12-21T15:48:53.581 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1550: wait_for_clean: local cur_active_clean 2019-12-21T15:48:53.581 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1551: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2019-12-21T15:48:53.582 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1551: wait_for_clean: get_timeout_delays 90 .1 2019-12-21T15:48:53.582 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1494: get_timeout_delays: shopt -q -o xtrace 2019-12-21T15:48:53.582 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1494: get_timeout_delays: echo true 2019-12-21T15:48:53.583 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1494: get_timeout_delays: local trace=true 2019-12-21T15:48:53.583 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1495: get_timeout_delays: true 2019-12-21T15:48:53.583 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1495: get_timeout_delays: shopt -u -o xtrace 2019-12-21T15:48:53.669 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1551: wait_for_clean: local -a delays 2019-12-21T15:48:53.670 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1552: wait_for_clean: local -i loop=0 2019-12-21T15:48:53.670 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1554: wait_for_clean: flush_pg_stats 2019-12-21T15:48:53.670 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1977: flush_pg_stats: local timeout=300 2019-12-21T15:48:53.670 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1979: flush_pg_stats: ceph osd ls 2019-12-21T15:48:53.942 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1979: flush_pg_stats: ids='0 2019-12-21T15:48:53.942 INFO:tasks.workunit.client.0.smithi172.stderr:1 2019-12-21T15:48:53.942 INFO:tasks.workunit.client.0.smithi172.stderr:2 2019-12-21T15:48:53.942 INFO:tasks.workunit.client.0.smithi172.stderr:3 2019-12-21T15:48:53.943 INFO:tasks.workunit.client.0.smithi172.stderr:4 2019-12-21T15:48:53.943 INFO:tasks.workunit.client.0.smithi172.stderr:5' 2019-12-21T15:48:53.943 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1980: flush_pg_stats: seqs= 2019-12-21T15:48:53.943 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: flush_pg_stats: for osd in $ids 2019-12-21T15:48:53.943 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2019-12-21T15:48:54.088 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: seq=21474836490 2019-12-21T15:48:54.088 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: flush_pg_stats: seqs=' 0-21474836490' 2019-12-21T15:48:54.088 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: flush_pg_stats: for osd in $ids 2019-12-21T15:48:54.089 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2019-12-21T15:48:54.208 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: seq=42949672968 2019-12-21T15:48:54.209 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: flush_pg_stats: seqs=' 0-21474836490 1-42949672968' 2019-12-21T15:48:54.209 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: flush_pg_stats: for osd in $ids 2019-12-21T15:48:54.209 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2019-12-21T15:48:54.326 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: seq=64424509447 2019-12-21T15:48:54.326 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: flush_pg_stats: seqs=' 0-21474836490 1-42949672968 2-64424509447' 2019-12-21T15:48:54.326 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: flush_pg_stats: for osd in $ids 2019-12-21T15:48:54.327 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2019-12-21T15:48:54.445 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: seq=85899345925 2019-12-21T15:48:54.445 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: flush_pg_stats: seqs=' 0-21474836490 1-42949672968 2-64424509447 3-85899345925' 2019-12-21T15:48:54.445 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: flush_pg_stats: for osd in $ids 2019-12-21T15:48:54.445 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2019-12-21T15:48:54.563 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: seq=107374182405 2019-12-21T15:48:54.563 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: flush_pg_stats: seqs=' 0-21474836490 1-42949672968 2-64424509447 3-85899345925 4-107374182405' 2019-12-21T15:48:54.563 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: flush_pg_stats: for osd in $ids 2019-12-21T15:48:54.564 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2019-12-21T15:48:54.683 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: seq=128849018884 2019-12-21T15:48:54.683 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: flush_pg_stats: seqs=' 0-21474836490 1-42949672968 2-64424509447 3-85899345925 4-107374182405 5-128849018884' 2019-12-21T15:48:54.683 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: flush_pg_stats: for s in $seqs 2019-12-21T15:48:54.684 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: echo 0-21474836490 2019-12-21T15:48:54.684 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: cut -d - -f 1 2019-12-21T15:48:54.685 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: osd=0 2019-12-21T15:48:54.686 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: echo 0-21474836490 2019-12-21T15:48:54.686 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: cut -d - -f 2 2019-12-21T15:48:54.687 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: seq=21474836490 2019-12-21T15:48:54.687 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1989: flush_pg_stats: echo 'waiting osd.0 seq 21474836490' 2019-12-21T15:48:54.687 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.0 seq 21474836490 2019-12-21T15:48:54.688 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 0 2019-12-21T15:48:54.922 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 21474836490 -lt 21474836490 2019-12-21T15:48:54.922 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: flush_pg_stats: for s in $seqs 2019-12-21T15:48:54.923 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: echo 1-42949672968 2019-12-21T15:48:54.923 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: cut -d - -f 1 2019-12-21T15:48:54.923 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: osd=1 2019-12-21T15:48:54.924 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: echo 1-42949672968 2019-12-21T15:48:54.925 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: cut -d - -f 2 2019-12-21T15:48:54.925 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: seq=42949672968 2019-12-21T15:48:54.926 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1989: flush_pg_stats: echo 'waiting osd.1 seq 42949672968' 2019-12-21T15:48:54.926 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.1 seq 42949672968 2019-12-21T15:48:54.926 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 1 2019-12-21T15:48:55.169 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 42949672968 -lt 42949672968 2019-12-21T15:48:55.170 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: flush_pg_stats: for s in $seqs 2019-12-21T15:48:55.170 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: echo 2-64424509447 2019-12-21T15:48:55.170 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: cut -d - -f 1 2019-12-21T15:48:55.170 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: osd=2 2019-12-21T15:48:55.170 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: echo 2-64424509447 2019-12-21T15:48:55.170 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: cut -d - -f 2 2019-12-21T15:48:55.171 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: seq=64424509447 2019-12-21T15:48:55.171 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1989: flush_pg_stats: echo 'waiting osd.2 seq 64424509447' 2019-12-21T15:48:55.171 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 2 2019-12-21T15:48:55.171 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.2 seq 64424509447 2019-12-21T15:48:55.402 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 64424509447 -lt 64424509447 2019-12-21T15:48:55.402 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: flush_pg_stats: for s in $seqs 2019-12-21T15:48:55.402 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: echo 3-85899345925 2019-12-21T15:48:55.404 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: cut -d - -f 1 2019-12-21T15:48:55.404 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: osd=3 2019-12-21T15:48:55.404 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: echo 3-85899345925 2019-12-21T15:48:55.405 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: cut -d - -f 2 2019-12-21T15:48:55.406 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.3 seq 85899345925 2019-12-21T15:48:55.406 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: seq=85899345925 2019-12-21T15:48:55.406 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1989: flush_pg_stats: echo 'waiting osd.3 seq 85899345925' 2019-12-21T15:48:55.406 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 3 2019-12-21T15:48:55.640 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 85899345925 -lt 85899345925 2019-12-21T15:48:55.640 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: flush_pg_stats: for s in $seqs 2019-12-21T15:48:55.641 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: echo 4-107374182405 2019-12-21T15:48:55.641 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: cut -d - -f 1 2019-12-21T15:48:55.642 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: osd=4 2019-12-21T15:48:55.643 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: echo 4-107374182405 2019-12-21T15:48:55.643 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: cut -d - -f 2 2019-12-21T15:48:55.644 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: seq=107374182405 2019-12-21T15:48:55.644 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1989: flush_pg_stats: echo 'waiting osd.4 seq 107374182405' 2019-12-21T15:48:55.644 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.4 seq 107374182405 2019-12-21T15:48:55.645 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 4 2019-12-21T15:48:55.883 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 107374182404 -lt 107374182405 2019-12-21T15:48:55.884 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1991: flush_pg_stats: sleep 1 2019-12-21T15:48:56.884 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1992: flush_pg_stats: '[' 300 -eq 0 ']' 2019-12-21T15:48:56.885 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 4 2019-12-21T15:48:57.130 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 107374182405 -lt 107374182405 2019-12-21T15:48:57.130 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: flush_pg_stats: for s in $seqs 2019-12-21T15:48:57.131 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: echo 5-128849018884 2019-12-21T15:48:57.131 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: cut -d - -f 1 2019-12-21T15:48:57.132 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: osd=5 2019-12-21T15:48:57.133 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: echo 5-128849018884 2019-12-21T15:48:57.133 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: cut -d - -f 2 2019-12-21T15:48:57.134 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: seq=128849018884 2019-12-21T15:48:57.134 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1989: flush_pg_stats: echo 'waiting osd.5 seq 128849018884' 2019-12-21T15:48:57.134 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.5 seq 128849018884 2019-12-21T15:48:57.135 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 5 2019-12-21T15:48:57.372 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 128849018884 -lt 128849018884 2019-12-21T15:48:57.373 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1555: wait_for_clean: get_num_pgs 2019-12-21T15:48:57.373 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: ceph --format json status 2019-12-21T15:48:57.373 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: jq .pgmap.num_pgs 2019-12-21T15:48:57.656 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1555: wait_for_clean: test 48 == 0 2019-12-21T15:48:57.656 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1559: wait_for_clean: true 2019-12-21T15:48:57.657 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1563: wait_for_clean: get_num_active_clean 2019-12-21T15:48:57.657 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: get_num_active_clean: local expression 2019-12-21T15:48:57.658 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2019-12-21T15:48:57.658 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("stale") | not)' 2019-12-21T15:48:57.658 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: ceph --format json pg dump pgs 2019-12-21T15:48:57.658 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2019-12-21T15:48:57.891 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1563: wait_for_clean: cur_active_clean=48 2019-12-21T15:48:57.892 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1564: wait_for_clean: get_num_pgs 2019-12-21T15:48:57.892 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: ceph --format json status 2019-12-21T15:48:57.892 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: jq .pgmap.num_pgs 2019-12-21T15:48:58.171 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1564: wait_for_clean: test 48 = 48 2019-12-21T15:48:58.171 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1564: wait_for_clean: break 2019-12-21T15:48:58.171 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: return 0 2019-12-21T15:48:58.171 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:71: TEST_recover_unexpected: dd if=/dev/urandom of=datafile bs=4k count=2 2019-12-21T15:48:58.172 INFO:tasks.workunit.client.0.smithi172.stderr:2+0 records in 2019-12-21T15:48:58.172 INFO:tasks.workunit.client.0.smithi172.stderr:2+0 records out 2019-12-21T15:48:58.172 INFO:tasks.workunit.client.0.smithi172.stderr:8192 bytes (8.2 kB, 8.0 KiB) copied, 7.3304e-05 s, 112 MB/s 2019-12-21T15:48:58.173 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:72: TEST_recover_unexpected: seq 1 3 2019-12-21T15:48:58.173 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:72: TEST_recover_unexpected: for i in $(seq 1 $POOLS) 2019-12-21T15:48:58.173 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: seq 1 1000 2019-12-21T15:48:58.174 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:48:58.175 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj1 datafile 2019-12-21T15:48:58.221 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:48:58.221 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj2 datafile 2019-12-21T15:48:58.340 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:48:58.340 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj3 datafile 2019-12-21T15:48:58.440 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:48:58.441 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj4 datafile 2019-12-21T15:48:58.490 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:48:58.490 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj5 datafile 2019-12-21T15:48:58.531 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:48:58.532 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj6 datafile 2019-12-21T15:48:58.649 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:48:58.650 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj7 datafile 2019-12-21T15:48:58.719 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:48:58.719 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj8 datafile 2019-12-21T15:48:58.760 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:48:58.760 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj9 datafile 2019-12-21T15:48:58.796 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:48:58.796 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj10 datafile 2019-12-21T15:48:58.848 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:48:58.848 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj11 datafile 2019-12-21T15:48:58.933 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:48:58.934 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj12 datafile 2019-12-21T15:48:58.977 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:48:58.977 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj13 datafile 2019-12-21T15:48:59.035 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:48:59.035 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj14 datafile 2019-12-21T15:48:59.073 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:48:59.073 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj15 datafile 2019-12-21T15:48:59.184 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:48:59.184 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj16 datafile 2019-12-21T15:48:59.245 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:48:59.245 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj17 datafile 2019-12-21T15:48:59.297 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:48:59.298 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj18 datafile 2019-12-21T15:48:59.340 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:48:59.340 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj19 datafile 2019-12-21T15:48:59.435 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:48:59.435 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj20 datafile 2019-12-21T15:48:59.481 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:48:59.481 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj21 datafile 2019-12-21T15:48:59.560 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:48:59.560 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj22 datafile 2019-12-21T15:48:59.661 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:48:59.661 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj23 datafile 2019-12-21T15:48:59.706 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:48:59.706 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj24 datafile 2019-12-21T15:48:59.756 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:48:59.757 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj25 datafile 2019-12-21T15:48:59.798 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:48:59.798 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj26 datafile 2019-12-21T15:48:59.870 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:48:59.870 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj27 datafile 2019-12-21T15:48:59.928 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:48:59.928 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj28 datafile 2019-12-21T15:49:00.003 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:00.004 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj29 datafile 2019-12-21T15:49:00.087 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:00.088 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj30 datafile 2019-12-21T15:49:00.132 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:00.132 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj31 datafile 2019-12-21T15:49:00.176 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:00.177 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj32 datafile 2019-12-21T15:49:00.254 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:00.254 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj33 datafile 2019-12-21T15:49:00.298 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:00.299 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj34 datafile 2019-12-21T15:49:00.380 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:00.380 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj35 datafile 2019-12-21T15:49:00.443 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:00.443 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj36 datafile 2019-12-21T15:49:00.482 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:00.482 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj37 datafile 2019-12-21T15:49:00.555 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:00.555 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj38 datafile 2019-12-21T15:49:00.632 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:00.632 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj39 datafile 2019-12-21T15:49:00.723 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:00.723 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj40 datafile 2019-12-21T15:49:00.765 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:00.766 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj41 datafile 2019-12-21T15:49:00.840 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:00.840 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj42 datafile 2019-12-21T15:49:00.885 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:00.885 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj43 datafile 2019-12-21T15:49:00.926 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:00.927 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj44 datafile 2019-12-21T15:49:00.965 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:00.965 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj45 datafile 2019-12-21T15:49:01.041 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:01.041 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj46 datafile 2019-12-21T15:49:01.076 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:01.076 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj47 datafile 2019-12-21T15:49:01.199 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:01.200 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj48 datafile 2019-12-21T15:49:01.300 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:01.300 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj49 datafile 2019-12-21T15:49:01.338 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:01.338 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj50 datafile 2019-12-21T15:49:01.374 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:01.374 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj51 datafile 2019-12-21T15:49:01.458 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:01.459 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj52 datafile 2019-12-21T15:49:01.498 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:01.499 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj53 datafile 2019-12-21T15:49:01.535 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:01.535 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj54 datafile 2019-12-21T15:49:01.574 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:01.575 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj55 datafile 2019-12-21T15:49:01.659 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:01.659 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj56 datafile 2019-12-21T15:49:01.697 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:01.698 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj57 datafile 2019-12-21T15:49:01.740 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:01.740 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj58 datafile 2019-12-21T15:49:01.789 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:01.789 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj59 datafile 2019-12-21T15:49:01.857 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:01.857 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj60 datafile 2019-12-21T15:49:01.902 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:01.902 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj61 datafile 2019-12-21T15:49:01.984 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:01.985 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj62 datafile 2019-12-21T15:49:02.102 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:02.102 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj63 datafile 2019-12-21T15:49:02.193 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:02.194 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj64 datafile 2019-12-21T15:49:02.240 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:02.241 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj65 datafile 2019-12-21T15:49:02.282 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:02.282 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj66 datafile 2019-12-21T15:49:02.327 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:02.327 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj67 datafile 2019-12-21T15:49:02.411 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:02.411 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj68 datafile 2019-12-21T15:49:02.457 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:02.457 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj69 datafile 2019-12-21T15:49:02.498 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:02.499 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj70 datafile 2019-12-21T15:49:02.555 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:02.555 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj71 datafile 2019-12-21T15:49:02.611 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:02.611 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj72 datafile 2019-12-21T15:49:02.703 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:02.704 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj73 datafile 2019-12-21T15:49:02.779 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:02.779 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj74 datafile 2019-12-21T15:49:02.863 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:02.863 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj75 datafile 2019-12-21T15:49:02.954 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:02.955 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj76 datafile 2019-12-21T15:49:03.000 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:03.000 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj77 datafile 2019-12-21T15:49:03.080 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:03.080 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj78 datafile 2019-12-21T15:49:03.157 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:03.158 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj79 datafile 2019-12-21T15:49:03.199 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:03.199 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj80 datafile 2019-12-21T15:49:03.261 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:03.261 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj81 datafile 2019-12-21T15:49:03.307 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:03.308 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj82 datafile 2019-12-21T15:49:03.344 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:03.344 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj83 datafile 2019-12-21T15:49:03.439 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:03.439 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj84 datafile 2019-12-21T15:49:03.514 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:03.514 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj85 datafile 2019-12-21T15:49:03.561 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:03.561 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj86 datafile 2019-12-21T15:49:03.604 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:03.604 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj87 datafile 2019-12-21T15:49:03.681 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:03.682 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj88 datafile 2019-12-21T15:49:03.732 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:03.732 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj89 datafile 2019-12-21T15:49:03.816 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:03.816 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj90 datafile 2019-12-21T15:49:03.860 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:03.861 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj91 datafile 2019-12-21T15:49:03.902 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:03.902 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj92 datafile 2019-12-21T15:49:04.008 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:04.008 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj93 datafile 2019-12-21T15:49:04.046 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:04.047 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj94 datafile 2019-12-21T15:49:04.081 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:04.082 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj95 datafile 2019-12-21T15:49:04.124 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:04.124 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj96 datafile 2019-12-21T15:49:04.161 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:04.161 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj97 datafile 2019-12-21T15:49:04.203 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:04.203 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj98 datafile 2019-12-21T15:49:04.241 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:04.241 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj99 datafile 2019-12-21T15:49:04.341 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:04.342 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj100 datafile 2019-12-21T15:49:04.451 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:04.451 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj101 datafile 2019-12-21T15:49:04.499 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:04.499 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj102 datafile 2019-12-21T15:49:04.536 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:04.536 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj103 datafile 2019-12-21T15:49:04.619 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:04.620 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj104 datafile 2019-12-21T15:49:04.672 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:04.672 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj105 datafile 2019-12-21T15:49:04.785 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:04.785 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj106 datafile 2019-12-21T15:49:04.824 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:04.825 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj107 datafile 2019-12-21T15:49:04.868 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:04.868 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj108 datafile 2019-12-21T15:49:04.908 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:04.908 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj109 datafile 2019-12-21T15:49:04.977 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:04.977 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj110 datafile 2019-12-21T15:49:05.017 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:05.017 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj111 datafile 2019-12-21T15:49:05.077 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:05.077 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj112 datafile 2019-12-21T15:49:05.116 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:05.116 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj113 datafile 2019-12-21T15:49:05.158 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:05.158 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj114 datafile 2019-12-21T15:49:05.203 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:05.203 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj115 datafile 2019-12-21T15:49:05.311 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:05.311 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj116 datafile 2019-12-21T15:49:05.420 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:05.420 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj117 datafile 2019-12-21T15:49:05.466 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:05.466 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj118 datafile 2019-12-21T15:49:05.511 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:05.512 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj119 datafile 2019-12-21T15:49:05.595 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:05.595 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj120 datafile 2019-12-21T15:49:05.696 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:05.696 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj121 datafile 2019-12-21T15:49:05.732 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:05.732 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj122 datafile 2019-12-21T15:49:05.768 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:05.769 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj123 datafile 2019-12-21T15:49:05.808 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:05.808 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj124 datafile 2019-12-21T15:49:05.844 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:05.844 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj125 datafile 2019-12-21T15:49:05.883 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:05.883 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj126 datafile 2019-12-21T15:49:05.929 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:05.929 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj127 datafile 2019-12-21T15:49:05.979 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:05.980 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj128 datafile 2019-12-21T15:49:06.089 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:06.089 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj129 datafile 2019-12-21T15:49:06.197 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:06.198 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj130 datafile 2019-12-21T15:49:06.232 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:06.232 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj131 datafile 2019-12-21T15:49:06.275 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:06.275 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj132 datafile 2019-12-21T15:49:06.316 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:06.317 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj133 datafile 2019-12-21T15:49:06.358 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:06.358 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj134 datafile 2019-12-21T15:49:06.413 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:06.413 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj135 datafile 2019-12-21T15:49:06.450 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:06.450 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj136 datafile 2019-12-21T15:49:06.500 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:06.500 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj137 datafile 2019-12-21T15:49:06.614 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:06.615 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj138 datafile 2019-12-21T15:49:06.649 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:06.649 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj139 datafile 2019-12-21T15:49:06.732 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:06.732 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj140 datafile 2019-12-21T15:49:06.832 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:06.832 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj141 datafile 2019-12-21T15:49:06.916 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:06.916 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj142 datafile 2019-12-21T15:49:06.954 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:06.954 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj143 datafile 2019-12-21T15:49:07.041 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:07.041 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj144 datafile 2019-12-21T15:49:07.079 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:07.079 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj145 datafile 2019-12-21T15:49:07.116 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:07.116 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj146 datafile 2019-12-21T15:49:07.167 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:07.167 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj147 datafile 2019-12-21T15:49:07.208 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:07.209 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj148 datafile 2019-12-21T15:49:07.250 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:07.250 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj149 datafile 2019-12-21T15:49:07.333 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:07.333 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj150 datafile 2019-12-21T15:49:07.371 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:07.371 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj151 datafile 2019-12-21T15:49:07.469 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:07.469 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj152 datafile 2019-12-21T15:49:07.559 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:07.559 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj153 datafile 2019-12-21T15:49:07.636 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:07.636 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj154 datafile 2019-12-21T15:49:07.683 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:07.684 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj155 datafile 2019-12-21T15:49:07.724 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:07.724 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj156 datafile 2019-12-21T15:49:07.760 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:07.760 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj157 datafile 2019-12-21T15:49:07.826 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:07.827 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj158 datafile 2019-12-21T15:49:07.867 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:07.867 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj159 datafile 2019-12-21T15:49:07.944 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:07.944 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj160 datafile 2019-12-21T15:49:07.979 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:07.979 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj161 datafile 2019-12-21T15:49:08.069 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:08.069 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj162 datafile 2019-12-21T15:49:08.125 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:08.125 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj163 datafile 2019-12-21T15:49:08.184 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:08.184 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj164 datafile 2019-12-21T15:49:08.286 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:08.286 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj165 datafile 2019-12-21T15:49:08.321 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:08.321 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj166 datafile 2019-12-21T15:49:08.364 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:08.364 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj167 datafile 2019-12-21T15:49:08.453 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:08.453 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj168 datafile 2019-12-21T15:49:08.571 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:08.571 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj169 datafile 2019-12-21T15:49:08.625 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:08.626 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj170 datafile 2019-12-21T15:49:08.721 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:08.721 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj171 datafile 2019-12-21T15:49:08.775 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:08.776 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj172 datafile 2019-12-21T15:49:08.887 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:08.887 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj173 datafile 2019-12-21T15:49:08.939 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:08.939 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj174 datafile 2019-12-21T15:49:09.023 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:09.023 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj175 datafile 2019-12-21T15:49:09.067 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:09.067 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj176 datafile 2019-12-21T15:49:09.148 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:09.148 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj177 datafile 2019-12-21T15:49:09.182 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:09.183 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj178 datafile 2019-12-21T15:49:09.223 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:09.223 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj179 datafile 2019-12-21T15:49:09.332 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:09.332 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj180 datafile 2019-12-21T15:49:09.416 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:09.416 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj181 datafile 2019-12-21T15:49:09.459 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:09.459 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj182 datafile 2019-12-21T15:49:09.492 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:09.493 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj183 datafile 2019-12-21T15:49:09.534 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:09.534 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj184 datafile 2019-12-21T15:49:09.598 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:09.599 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj185 datafile 2019-12-21T15:49:09.691 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:09.692 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj186 datafile 2019-12-21T15:49:09.730 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:09.730 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj187 datafile 2019-12-21T15:49:09.808 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:09.809 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj188 datafile 2019-12-21T15:49:09.842 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:09.842 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj189 datafile 2019-12-21T15:49:09.881 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:09.881 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj190 datafile 2019-12-21T15:49:09.959 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:09.959 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj191 datafile 2019-12-21T15:49:09.996 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:09.997 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj192 datafile 2019-12-21T15:49:10.034 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:10.034 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj193 datafile 2019-12-21T15:49:10.117 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:10.118 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj194 datafile 2019-12-21T15:49:10.211 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:10.211 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj195 datafile 2019-12-21T15:49:10.251 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:10.251 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj196 datafile 2019-12-21T15:49:10.305 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:10.305 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj197 datafile 2019-12-21T15:49:10.394 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:10.394 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj198 datafile 2019-12-21T15:49:10.439 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:10.439 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj199 datafile 2019-12-21T15:49:10.481 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:10.481 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj200 datafile 2019-12-21T15:49:10.516 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:10.517 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj201 datafile 2019-12-21T15:49:10.636 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:10.636 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj202 datafile 2019-12-21T15:49:10.681 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:10.681 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj203 datafile 2019-12-21T15:49:10.717 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:10.717 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj204 datafile 2019-12-21T15:49:10.756 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:10.756 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj205 datafile 2019-12-21T15:49:10.853 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:10.853 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj206 datafile 2019-12-21T15:49:10.891 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:10.891 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj207 datafile 2019-12-21T15:49:10.931 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:10.932 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj208 datafile 2019-12-21T15:49:10.976 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:10.976 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj209 datafile 2019-12-21T15:49:11.070 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:11.071 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj210 datafile 2019-12-21T15:49:11.117 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:11.118 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj211 datafile 2019-12-21T15:49:11.159 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:11.159 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj212 datafile 2019-12-21T15:49:11.220 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:11.220 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj213 datafile 2019-12-21T15:49:11.258 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:11.258 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj214 datafile 2019-12-21T15:49:11.354 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:11.354 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj215 datafile 2019-12-21T15:49:11.398 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:11.398 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj216 datafile 2019-12-21T15:49:11.442 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:11.443 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj217 datafile 2019-12-21T15:49:11.521 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:11.521 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj218 datafile 2019-12-21T15:49:11.579 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:11.579 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj219 datafile 2019-12-21T15:49:11.618 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:11.618 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj220 datafile 2019-12-21T15:49:11.730 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:11.730 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj221 datafile 2019-12-21T15:49:11.805 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:11.806 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj222 datafile 2019-12-21T15:49:11.889 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:11.889 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj223 datafile 2019-12-21T15:49:11.925 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:11.925 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj224 datafile 2019-12-21T15:49:12.006 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:12.006 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj225 datafile 2019-12-21T15:49:12.048 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:12.048 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj226 datafile 2019-12-21T15:49:12.109 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:12.109 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj227 datafile 2019-12-21T15:49:12.151 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:12.151 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj228 datafile 2019-12-21T15:49:12.194 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:12.194 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj229 datafile 2019-12-21T15:49:12.235 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:12.236 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj230 datafile 2019-12-21T15:49:12.315 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:12.315 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj231 datafile 2019-12-21T15:49:12.359 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:12.360 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj232 datafile 2019-12-21T15:49:12.482 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:12.482 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj233 datafile 2019-12-21T15:49:12.558 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:12.559 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj234 datafile 2019-12-21T15:49:12.627 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:12.627 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj235 datafile 2019-12-21T15:49:12.708 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:12.709 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj236 datafile 2019-12-21T15:49:12.743 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:12.743 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj237 datafile 2019-12-21T15:49:12.784 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:12.785 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj238 datafile 2019-12-21T15:49:12.827 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:12.827 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj239 datafile 2019-12-21T15:49:12.868 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:12.868 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj240 datafile 2019-12-21T15:49:12.950 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:12.950 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj241 datafile 2019-12-21T15:49:12.993 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:12.993 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj242 datafile 2019-12-21T15:49:13.043 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:13.043 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj243 datafile 2019-12-21T15:49:13.176 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:13.176 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj244 datafile 2019-12-21T15:49:13.260 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:13.260 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj245 datafile 2019-12-21T15:49:13.301 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:13.302 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj246 datafile 2019-12-21T15:49:13.343 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:13.343 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj247 datafile 2019-12-21T15:49:13.537 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:13.537 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj248 datafile 2019-12-21T15:49:13.631 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:13.632 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj249 datafile 2019-12-21T15:49:13.754 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:13.754 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj250 datafile 2019-12-21T15:49:13.928 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:13.929 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj251 datafile 2019-12-21T15:49:13.973 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:13.973 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj252 datafile 2019-12-21T15:49:14.069 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:14.069 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj253 datafile 2019-12-21T15:49:14.153 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:14.153 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj254 datafile 2019-12-21T15:49:14.197 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:14.197 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj255 datafile 2019-12-21T15:49:14.231 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:14.231 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj256 datafile 2019-12-21T15:49:14.286 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:14.287 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj257 datafile 2019-12-21T15:49:14.371 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:14.371 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj258 datafile 2019-12-21T15:49:14.446 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:14.446 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj259 datafile 2019-12-21T15:49:14.484 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:14.484 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj260 datafile 2019-12-21T15:49:14.523 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:14.523 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj261 datafile 2019-12-21T15:49:14.630 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:14.630 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj262 datafile 2019-12-21T15:49:14.668 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:14.668 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj263 datafile 2019-12-21T15:49:14.710 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:14.710 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj264 datafile 2019-12-21T15:49:14.748 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:14.748 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj265 datafile 2019-12-21T15:49:14.839 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:14.839 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj266 datafile 2019-12-21T15:49:14.881 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:14.881 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj267 datafile 2019-12-21T15:49:14.939 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:14.939 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj268 datafile 2019-12-21T15:49:14.985 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:14.985 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj269 datafile 2019-12-21T15:49:15.056 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:15.057 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj270 datafile 2019-12-21T15:49:15.102 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:15.102 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj271 datafile 2019-12-21T15:49:15.140 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:15.140 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj272 datafile 2019-12-21T15:49:15.248 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:15.248 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj273 datafile 2019-12-21T15:49:15.294 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:15.294 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj274 datafile 2019-12-21T15:49:15.382 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:15.382 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj275 datafile 2019-12-21T15:49:15.423 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:15.424 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj276 datafile 2019-12-21T15:49:15.460 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:15.461 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj277 datafile 2019-12-21T15:49:15.502 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:15.502 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj278 datafile 2019-12-21T15:49:15.546 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:15.546 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj279 datafile 2019-12-21T15:49:15.633 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:15.633 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj280 datafile 2019-12-21T15:49:15.669 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:15.669 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj281 datafile 2019-12-21T15:49:15.767 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:15.767 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj282 datafile 2019-12-21T15:49:15.851 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:15.851 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj283 datafile 2019-12-21T15:49:15.894 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:15.894 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj284 datafile 2019-12-21T15:49:15.940 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:15.941 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj285 datafile 2019-12-21T15:49:15.976 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:15.976 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj286 datafile 2019-12-21T15:49:16.016 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:16.016 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj287 datafile 2019-12-21T15:49:16.061 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:16.061 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj288 datafile 2019-12-21T15:49:16.111 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:16.111 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj289 datafile 2019-12-21T15:49:16.153 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:16.153 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj290 datafile 2019-12-21T15:49:16.251 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:16.251 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj291 datafile 2019-12-21T15:49:16.335 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:16.335 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj292 datafile 2019-12-21T15:49:16.418 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:16.419 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj293 datafile 2019-12-21T15:49:16.456 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:16.457 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj294 datafile 2019-12-21T15:49:16.527 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:16.527 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj295 datafile 2019-12-21T15:49:16.571 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:16.571 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj296 datafile 2019-12-21T15:49:16.636 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:16.636 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj297 datafile 2019-12-21T15:49:16.719 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:16.719 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj298 datafile 2019-12-21T15:49:16.803 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:16.803 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj299 datafile 2019-12-21T15:49:16.845 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:16.845 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj300 datafile 2019-12-21T15:49:16.878 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:16.878 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj301 datafile 2019-12-21T15:49:16.945 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:16.945 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj302 datafile 2019-12-21T15:49:16.986 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:16.986 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj303 datafile 2019-12-21T15:49:17.071 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:17.071 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj304 datafile 2019-12-21T15:49:17.155 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:17.155 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj305 datafile 2019-12-21T15:49:17.255 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:17.255 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj306 datafile 2019-12-21T15:49:17.295 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:17.295 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj307 datafile 2019-12-21T15:49:17.345 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:17.345 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj308 datafile 2019-12-21T15:49:17.422 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:17.422 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj309 datafile 2019-12-21T15:49:17.470 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:17.470 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj310 datafile 2019-12-21T15:49:17.506 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:17.507 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj311 datafile 2019-12-21T15:49:17.545 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:17.545 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj312 datafile 2019-12-21T15:49:17.578 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:17.578 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj313 datafile 2019-12-21T15:49:17.656 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:17.656 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj314 datafile 2019-12-21T15:49:17.714 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:17.715 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj315 datafile 2019-12-21T15:49:17.807 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:17.807 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj316 datafile 2019-12-21T15:49:17.849 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:17.850 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj317 datafile 2019-12-21T15:49:17.886 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:17.887 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj318 datafile 2019-12-21T15:49:17.965 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:17.966 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj319 datafile 2019-12-21T15:49:18.076 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:18.076 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj320 datafile 2019-12-21T15:49:18.120 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:18.120 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj321 datafile 2019-12-21T15:49:18.162 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:18.162 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj322 datafile 2019-12-21T15:49:18.203 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:18.204 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj323 datafile 2019-12-21T15:49:18.244 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:18.245 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj324 datafile 2019-12-21T15:49:18.324 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:18.325 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj325 datafile 2019-12-21T15:49:18.425 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:18.425 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj326 datafile 2019-12-21T15:49:18.461 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:18.462 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj327 datafile 2019-12-21T15:49:18.498 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:18.498 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj328 datafile 2019-12-21T15:49:18.533 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:18.534 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj329 datafile 2019-12-21T15:49:18.684 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:18.685 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj330 datafile 2019-12-21T15:49:18.751 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:18.751 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj331 datafile 2019-12-21T15:49:18.791 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:18.792 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj332 datafile 2019-12-21T15:49:18.837 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:18.837 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj333 datafile 2019-12-21T15:49:18.920 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:18.920 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj334 datafile 2019-12-21T15:49:18.956 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:18.957 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj335 datafile 2019-12-21T15:49:19.033 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:19.033 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj336 datafile 2019-12-21T15:49:19.072 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:19.072 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj337 datafile 2019-12-21T15:49:19.144 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:19.144 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj338 datafile 2019-12-21T15:49:19.247 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:19.248 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj339 datafile 2019-12-21T15:49:19.363 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:19.363 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj340 datafile 2019-12-21T15:49:19.403 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:19.403 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj341 datafile 2019-12-21T15:49:19.454 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:19.454 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj342 datafile 2019-12-21T15:49:19.493 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:19.493 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj343 datafile 2019-12-21T15:49:19.534 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:19.535 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj344 datafile 2019-12-21T15:49:19.576 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:19.576 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj345 datafile 2019-12-21T15:49:19.620 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:19.621 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj346 datafile 2019-12-21T15:49:19.728 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:19.729 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj347 datafile 2019-12-21T15:49:19.770 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:19.770 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj348 datafile 2019-12-21T15:49:19.804 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:19.804 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj349 datafile 2019-12-21T15:49:19.845 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:19.846 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj350 datafile 2019-12-21T15:49:19.920 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:19.921 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj351 datafile 2019-12-21T15:49:20.004 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:20.004 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj352 datafile 2019-12-21T15:49:20.113 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:20.114 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj353 datafile 2019-12-21T15:49:20.159 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:20.160 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj354 datafile 2019-12-21T15:49:20.204 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:20.204 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj355 datafile 2019-12-21T15:49:20.289 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:20.289 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj356 datafile 2019-12-21T15:49:20.326 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:20.327 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj357 datafile 2019-12-21T15:49:20.370 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:20.371 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj358 datafile 2019-12-21T15:49:20.412 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:20.412 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj359 datafile 2019-12-21T15:49:20.454 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:20.454 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj360 datafile 2019-12-21T15:49:20.493 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:20.493 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj361 datafile 2019-12-21T15:49:20.537 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:20.537 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj362 datafile 2019-12-21T15:49:20.602 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:20.602 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj363 datafile 2019-12-21T15:49:20.701 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:20.701 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj364 datafile 2019-12-21T15:49:20.773 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:20.774 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj365 datafile 2019-12-21T15:49:20.907 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:20.907 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj366 datafile 2019-12-21T15:49:20.942 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:20.942 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj367 datafile 2019-12-21T15:49:20.994 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:20.994 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj368 datafile 2019-12-21T15:49:21.066 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:21.066 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj369 datafile 2019-12-21T15:49:21.104 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:21.104 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj370 datafile 2019-12-21T15:49:21.183 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:21.183 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj371 datafile 2019-12-21T15:49:21.227 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:21.227 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj372 datafile 2019-12-21T15:49:21.270 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:21.270 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj373 datafile 2019-12-21T15:49:21.307 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:21.307 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj374 datafile 2019-12-21T15:49:21.346 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:21.346 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj375 datafile 2019-12-21T15:49:21.442 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:21.442 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj376 datafile 2019-12-21T15:49:21.485 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:21.485 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj377 datafile 2019-12-21T15:49:21.529 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:21.529 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj378 datafile 2019-12-21T15:49:21.579 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:21.579 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj379 datafile 2019-12-21T15:49:21.667 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:21.667 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj380 datafile 2019-12-21T15:49:21.704 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:21.704 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj381 datafile 2019-12-21T15:49:21.784 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:21.785 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj382 datafile 2019-12-21T15:49:21.829 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:21.829 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj383 datafile 2019-12-21T15:49:21.901 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:21.902 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj384 datafile 2019-12-21T15:49:21.952 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:21.952 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj385 datafile 2019-12-21T15:49:22.044 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:22.044 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj386 datafile 2019-12-21T15:49:22.087 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:22.088 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj387 datafile 2019-12-21T15:49:22.129 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:22.129 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj388 datafile 2019-12-21T15:49:22.211 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:22.211 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj389 datafile 2019-12-21T15:49:22.286 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:22.286 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj390 datafile 2019-12-21T15:49:22.327 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:22.327 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj391 datafile 2019-12-21T15:49:22.371 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:22.372 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj392 datafile 2019-12-21T15:49:22.470 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:22.470 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj393 datafile 2019-12-21T15:49:22.519 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:22.519 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj394 datafile 2019-12-21T15:49:22.563 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:22.563 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj395 datafile 2019-12-21T15:49:22.655 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:22.655 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj396 datafile 2019-12-21T15:49:22.690 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:22.691 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj397 datafile 2019-12-21T15:49:22.780 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:22.780 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj398 datafile 2019-12-21T15:49:22.818 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:22.819 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj399 datafile 2019-12-21T15:49:22.863 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:22.863 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj400 datafile 2019-12-21T15:49:22.910 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:22.911 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj401 datafile 2019-12-21T15:49:23.022 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:23.022 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj402 datafile 2019-12-21T15:49:23.063 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:23.063 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj403 datafile 2019-12-21T15:49:23.100 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:23.100 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj404 datafile 2019-12-21T15:49:23.197 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:23.198 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj405 datafile 2019-12-21T15:49:23.256 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:23.256 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj406 datafile 2019-12-21T15:49:23.297 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:23.297 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj407 datafile 2019-12-21T15:49:23.382 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:23.382 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj408 datafile 2019-12-21T15:49:23.418 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:23.419 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj409 datafile 2019-12-21T15:49:23.507 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:23.507 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj410 datafile 2019-12-21T15:49:23.546 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:23.547 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj411 datafile 2019-12-21T15:49:23.588 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:23.589 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj412 datafile 2019-12-21T15:49:23.709 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:23.709 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj413 datafile 2019-12-21T15:49:23.746 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:23.746 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj414 datafile 2019-12-21T15:49:23.799 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:23.799 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj415 datafile 2019-12-21T15:49:23.838 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:23.838 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj416 datafile 2019-12-21T15:49:23.885 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:23.886 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj417 datafile 2019-12-21T15:49:23.991 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:23.992 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj418 datafile 2019-12-21T15:49:24.175 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:24.175 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj419 datafile 2019-12-21T15:49:24.259 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:24.259 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj420 datafile 2019-12-21T15:49:24.302 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:24.302 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj421 datafile 2019-12-21T15:49:24.338 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:24.338 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj422 datafile 2019-12-21T15:49:24.380 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:24.380 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj423 datafile 2019-12-21T15:49:24.451 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:24.451 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj424 datafile 2019-12-21T15:49:24.535 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:24.535 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj425 datafile 2019-12-21T15:49:24.619 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:24.619 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj426 datafile 2019-12-21T15:49:24.669 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:24.669 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj427 datafile 2019-12-21T15:49:24.752 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:24.753 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj428 datafile 2019-12-21T15:49:24.828 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:24.828 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj429 datafile 2019-12-21T15:49:24.869 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:24.869 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj430 datafile 2019-12-21T15:49:24.904 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:24.905 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj431 datafile 2019-12-21T15:49:24.944 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:24.944 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj432 datafile 2019-12-21T15:49:24.983 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:24.983 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj433 datafile 2019-12-21T15:49:25.048 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:25.048 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj434 datafile 2019-12-21T15:49:25.097 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:25.097 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj435 datafile 2019-12-21T15:49:25.200 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:25.200 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj436 datafile 2019-12-21T15:49:25.254 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:25.254 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj437 datafile 2019-12-21T15:49:25.294 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:25.294 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj438 datafile 2019-12-21T15:49:25.388 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:25.388 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj439 datafile 2019-12-21T15:49:25.436 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:25.436 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj440 datafile 2019-12-21T15:49:25.513 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:25.514 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj441 datafile 2019-12-21T15:49:25.555 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:25.556 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj442 datafile 2019-12-21T15:49:25.639 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:25.639 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj443 datafile 2019-12-21T15:49:25.723 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:25.723 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj444 datafile 2019-12-21T15:49:25.781 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:25.781 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj445 datafile 2019-12-21T15:49:25.830 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:25.831 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj446 datafile 2019-12-21T15:49:25.869 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:25.869 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj447 datafile 2019-12-21T15:49:25.919 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:25.920 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj448 datafile 2019-12-21T15:49:25.960 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:25.961 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj449 datafile 2019-12-21T15:49:26.002 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:26.003 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj450 datafile 2019-12-21T15:49:26.115 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:26.115 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj451 datafile 2019-12-21T15:49:26.169 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:26.169 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj452 datafile 2019-12-21T15:49:26.249 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:26.249 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj453 datafile 2019-12-21T15:49:26.332 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:26.332 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj454 datafile 2019-12-21T15:49:26.378 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:26.378 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj455 datafile 2019-12-21T15:49:26.484 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:26.484 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj456 datafile 2019-12-21T15:49:26.527 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:26.528 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj457 datafile 2019-12-21T15:49:26.572 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:26.572 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj458 datafile 2019-12-21T15:49:26.644 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:26.644 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj459 datafile 2019-12-21T15:49:26.689 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:26.689 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj460 datafile 2019-12-21T15:49:26.775 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:26.775 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj461 datafile 2019-12-21T15:49:26.813 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:26.813 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj462 datafile 2019-12-21T15:49:26.892 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:26.893 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj463 datafile 2019-12-21T15:49:26.936 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:26.936 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj464 datafile 2019-12-21T15:49:26.971 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:26.972 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj465 datafile 2019-12-21T15:49:27.051 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:27.051 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj466 datafile 2019-12-21T15:49:27.135 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:27.135 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj467 datafile 2019-12-21T15:49:27.195 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:27.195 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj468 datafile 2019-12-21T15:49:27.277 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:27.277 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj469 datafile 2019-12-21T15:49:27.322 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:27.322 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj470 datafile 2019-12-21T15:49:27.361 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:27.361 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj471 datafile 2019-12-21T15:49:27.444 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:27.444 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj472 datafile 2019-12-21T15:49:27.528 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:27.528 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj473 datafile 2019-12-21T15:49:27.578 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:27.578 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj474 datafile 2019-12-21T15:49:27.622 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:27.623 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj475 datafile 2019-12-21T15:49:27.669 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:27.669 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj476 datafile 2019-12-21T15:49:27.719 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:27.720 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj477 datafile 2019-12-21T15:49:27.803 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:27.803 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj478 datafile 2019-12-21T15:49:27.847 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:27.848 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj479 datafile 2019-12-21T15:49:27.945 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:27.945 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj480 datafile 2019-12-21T15:49:28.037 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:28.038 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj481 datafile 2019-12-21T15:49:28.121 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:28.122 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj482 datafile 2019-12-21T15:49:28.157 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:28.158 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj483 datafile 2019-12-21T15:49:28.263 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:28.264 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj484 datafile 2019-12-21T15:49:28.303 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:28.303 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj485 datafile 2019-12-21T15:49:28.345 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:28.345 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj486 datafile 2019-12-21T15:49:28.390 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:28.390 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj487 datafile 2019-12-21T15:49:28.428 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:28.428 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj488 datafile 2019-12-21T15:49:28.514 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:28.514 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj489 datafile 2019-12-21T15:49:28.553 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:28.553 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj490 datafile 2019-12-21T15:49:28.656 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:28.656 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj491 datafile 2019-12-21T15:49:28.731 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:28.732 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj492 datafile 2019-12-21T15:49:28.770 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:28.770 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj493 datafile 2019-12-21T15:49:28.811 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:28.811 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj494 datafile 2019-12-21T15:49:28.898 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:28.898 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj495 datafile 2019-12-21T15:49:28.973 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:28.973 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj496 datafile 2019-12-21T15:49:29.057 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:29.058 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj497 datafile 2019-12-21T15:49:29.133 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:29.133 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj498 datafile 2019-12-21T15:49:29.173 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:29.173 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj499 datafile 2019-12-21T15:49:29.334 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:29.334 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj500 datafile 2019-12-21T15:49:29.372 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:29.372 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj501 datafile 2019-12-21T15:49:29.442 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:29.442 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj502 datafile 2019-12-21T15:49:29.487 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:29.487 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj503 datafile 2019-12-21T15:49:29.531 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:29.532 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj504 datafile 2019-12-21T15:49:29.573 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:29.573 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj505 datafile 2019-12-21T15:49:29.651 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:29.651 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj506 datafile 2019-12-21T15:49:29.735 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:29.735 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj507 datafile 2019-12-21T15:49:29.781 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:29.782 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj508 datafile 2019-12-21T15:49:29.823 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:29.823 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj509 datafile 2019-12-21T15:49:29.902 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:29.902 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj510 datafile 2019-12-21T15:49:30.019 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:30.019 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj511 datafile 2019-12-21T15:49:30.062 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:30.062 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj512 datafile 2019-12-21T15:49:30.107 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:30.107 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj513 datafile 2019-12-21T15:49:30.177 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:30.178 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj514 datafile 2019-12-21T15:49:30.224 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:30.224 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj515 datafile 2019-12-21T15:49:30.262 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:30.262 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj516 datafile 2019-12-21T15:49:30.307 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:30.307 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj517 datafile 2019-12-21T15:49:30.341 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:30.341 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj518 datafile 2019-12-21T15:49:30.412 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:30.412 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj519 datafile 2019-12-21T15:49:30.495 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:30.495 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj520 datafile 2019-12-21T15:49:30.574 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:30.574 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj521 datafile 2019-12-21T15:49:30.663 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:30.663 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj522 datafile 2019-12-21T15:49:30.718 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:30.719 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj523 datafile 2019-12-21T15:49:30.797 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:30.797 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj524 datafile 2019-12-21T15:49:30.837 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:30.837 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj525 datafile 2019-12-21T15:49:30.874 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:30.874 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj526 datafile 2019-12-21T15:49:30.907 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:30.907 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj527 datafile 2019-12-21T15:49:30.945 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:30.946 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj528 datafile 2019-12-21T15:49:30.987 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:30.988 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj529 datafile 2019-12-21T15:49:31.064 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:31.064 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj530 datafile 2019-12-21T15:49:31.099 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:31.099 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj531 datafile 2019-12-21T15:49:31.181 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:31.181 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj532 datafile 2019-12-21T15:49:31.273 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:31.273 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj533 datafile 2019-12-21T15:49:31.312 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:31.312 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj534 datafile 2019-12-21T15:49:31.354 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:31.354 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj535 datafile 2019-12-21T15:49:31.429 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:31.430 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj536 datafile 2019-12-21T15:49:31.474 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:31.474 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj537 datafile 2019-12-21T15:49:31.512 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:31.513 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj538 datafile 2019-12-21T15:49:31.615 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:31.615 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj539 datafile 2019-12-21T15:49:31.691 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:31.691 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj540 datafile 2019-12-21T15:49:31.737 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:31.737 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj541 datafile 2019-12-21T15:49:31.774 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:31.774 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj542 datafile 2019-12-21T15:49:31.849 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:31.850 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj543 datafile 2019-12-21T15:49:31.895 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:31.896 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj544 datafile 2019-12-21T15:49:31.941 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:31.941 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj545 datafile 2019-12-21T15:49:32.025 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:32.025 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj546 datafile 2019-12-21T15:49:32.121 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:32.121 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj547 datafile 2019-12-21T15:49:32.163 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:32.163 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj548 datafile 2019-12-21T15:49:32.247 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:32.247 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj549 datafile 2019-12-21T15:49:32.301 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:32.301 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj550 datafile 2019-12-21T15:49:32.354 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:32.355 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj551 datafile 2019-12-21T15:49:32.396 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:32.396 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj552 datafile 2019-12-21T15:49:32.438 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:32.438 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj553 datafile 2019-12-21T15:49:32.479 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:32.480 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj554 datafile 2019-12-21T15:49:32.585 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:32.585 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj555 datafile 2019-12-21T15:49:32.630 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:32.630 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj556 datafile 2019-12-21T15:49:32.688 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:32.688 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj557 datafile 2019-12-21T15:49:32.760 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:32.761 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj558 datafile 2019-12-21T15:49:32.869 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:32.869 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj559 datafile 2019-12-21T15:49:32.951 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:32.951 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj560 datafile 2019-12-21T15:49:33.011 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:33.012 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj561 datafile 2019-12-21T15:49:33.048 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:33.048 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj562 datafile 2019-12-21T15:49:33.082 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:33.082 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj563 datafile 2019-12-21T15:49:33.121 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:33.122 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj564 datafile 2019-12-21T15:49:33.163 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:33.164 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj565 datafile 2019-12-21T15:49:33.205 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:33.205 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj566 datafile 2019-12-21T15:49:33.288 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:33.288 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj567 datafile 2019-12-21T15:49:33.371 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:33.371 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj568 datafile 2019-12-21T15:49:33.480 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:33.480 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj569 datafile 2019-12-21T15:49:33.572 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:33.572 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj570 datafile 2019-12-21T15:49:33.610 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:33.610 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj571 datafile 2019-12-21T15:49:33.649 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:33.650 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj572 datafile 2019-12-21T15:49:33.706 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:33.706 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj573 datafile 2019-12-21T15:49:33.747 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:33.747 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj574 datafile 2019-12-21T15:49:33.789 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:33.789 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj575 datafile 2019-12-21T15:49:33.829 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:33.830 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj576 datafile 2019-12-21T15:49:33.872 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:33.872 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj577 datafile 2019-12-21T15:49:33.957 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:33.957 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj578 datafile 2019-12-21T15:49:34.015 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:34.015 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj579 datafile 2019-12-21T15:49:34.098 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:34.098 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj580 datafile 2019-12-21T15:49:34.147 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:34.147 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj581 datafile 2019-12-21T15:49:34.249 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:34.249 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj582 datafile 2019-12-21T15:49:34.289 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:34.289 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj583 datafile 2019-12-21T15:49:34.374 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:34.374 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj584 datafile 2019-12-21T15:49:34.458 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:34.458 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj585 datafile 2019-12-21T15:49:34.550 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:34.550 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj586 datafile 2019-12-21T15:49:34.667 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:34.667 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj587 datafile 2019-12-21T15:49:34.714 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:34.714 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj588 datafile 2019-12-21T15:49:34.793 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:34.793 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj589 datafile 2019-12-21T15:49:34.838 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:34.838 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj590 datafile 2019-12-21T15:49:34.880 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:34.881 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj591 datafile 2019-12-21T15:49:34.993 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:34.993 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj592 datafile 2019-12-21T15:49:35.039 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:35.039 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj593 datafile 2019-12-21T15:49:35.081 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:35.081 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj594 datafile 2019-12-21T15:49:35.177 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:35.177 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj595 datafile 2019-12-21T15:49:35.221 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:35.222 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj596 datafile 2019-12-21T15:49:35.302 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:35.302 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj597 datafile 2019-12-21T15:49:35.377 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:35.377 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj598 datafile 2019-12-21T15:49:35.421 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:35.422 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj599 datafile 2019-12-21T15:49:35.503 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:35.503 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj600 datafile 2019-12-21T15:49:35.546 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:35.546 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj601 datafile 2019-12-21T15:49:35.589 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:35.589 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj602 datafile 2019-12-21T15:49:35.656 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:35.656 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj603 datafile 2019-12-21T15:49:35.697 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:35.697 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj604 datafile 2019-12-21T15:49:35.739 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:35.739 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj605 datafile 2019-12-21T15:49:35.780 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:35.780 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj606 datafile 2019-12-21T15:49:35.862 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:35.862 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj607 datafile 2019-12-21T15:49:35.960 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:35.960 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj608 datafile 2019-12-21T15:49:36.046 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:36.046 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj609 datafile 2019-12-21T15:49:36.130 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:36.130 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj610 datafile 2019-12-21T15:49:36.173 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:36.173 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj611 datafile 2019-12-21T15:49:36.214 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:36.214 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj612 datafile 2019-12-21T15:49:36.256 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:36.256 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj613 datafile 2019-12-21T15:49:36.355 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:36.355 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj614 datafile 2019-12-21T15:49:36.439 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:36.439 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj615 datafile 2019-12-21T15:49:36.481 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:36.481 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj616 datafile 2019-12-21T15:49:36.523 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:36.523 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj617 datafile 2019-12-21T15:49:36.639 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:36.640 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj618 datafile 2019-12-21T15:49:36.698 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:36.698 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj619 datafile 2019-12-21T15:49:36.739 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:36.739 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj620 datafile 2019-12-21T15:49:36.848 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:36.849 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj621 datafile 2019-12-21T15:49:36.889 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:36.890 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj622 datafile 2019-12-21T15:49:36.924 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:36.925 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj623 datafile 2019-12-21T15:49:37.007 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:37.007 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj624 datafile 2019-12-21T15:49:37.055 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:37.056 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj625 datafile 2019-12-21T15:49:37.141 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:37.141 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj626 datafile 2019-12-21T15:49:37.180 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:37.181 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj627 datafile 2019-12-21T15:49:37.222 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:37.222 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj628 datafile 2019-12-21T15:49:37.283 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:37.283 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj629 datafile 2019-12-21T15:49:37.358 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:37.358 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj630 datafile 2019-12-21T15:49:37.397 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:37.397 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj631 datafile 2019-12-21T15:49:37.440 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:37.440 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj632 datafile 2019-12-21T15:49:37.481 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:37.481 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj633 datafile 2019-12-21T15:49:37.523 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:37.523 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj634 datafile 2019-12-21T15:49:37.566 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:37.566 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj635 datafile 2019-12-21T15:49:37.606 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:37.607 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj636 datafile 2019-12-21T15:49:37.709 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:37.709 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj637 datafile 2019-12-21T15:49:37.756 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:37.756 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj638 datafile 2019-12-21T15:49:37.834 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:37.834 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj639 datafile 2019-12-21T15:49:37.919 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:37.920 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj640 datafile 2019-12-21T15:49:38.001 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:38.002 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj641 datafile 2019-12-21T15:49:38.049 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:38.049 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj642 datafile 2019-12-21T15:49:38.144 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:38.144 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj643 datafile 2019-12-21T15:49:38.189 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:38.189 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj644 datafile 2019-12-21T15:49:38.278 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:38.278 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj645 datafile 2019-12-21T15:49:38.322 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:38.322 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj646 datafile 2019-12-21T15:49:38.365 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:38.366 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj647 datafile 2019-12-21T15:49:38.408 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:38.408 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj648 datafile 2019-12-21T15:49:38.466 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:38.466 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj649 datafile 2019-12-21T15:49:38.502 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:38.503 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj650 datafile 2019-12-21T15:49:38.548 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:38.548 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj651 datafile 2019-12-21T15:49:38.654 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:38.654 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj652 datafile 2019-12-21T15:49:38.726 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:38.726 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj653 datafile 2019-12-21T15:49:38.805 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:38.806 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj654 datafile 2019-12-21T15:49:38.888 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:38.888 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj655 datafile 2019-12-21T15:49:38.932 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:38.932 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj656 datafile 2019-12-21T15:49:39.047 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:39.047 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj657 datafile 2019-12-21T15:49:39.091 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:39.091 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj658 datafile 2019-12-21T15:49:39.133 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:39.133 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj659 datafile 2019-12-21T15:49:39.183 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:39.183 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj660 datafile 2019-12-21T15:49:39.224 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:39.225 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj661 datafile 2019-12-21T15:49:39.323 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:39.323 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj662 datafile 2019-12-21T15:49:39.398 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:39.398 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj663 datafile 2019-12-21T15:49:39.444 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:39.444 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj664 datafile 2019-12-21T15:49:39.494 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:39.494 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj665 datafile 2019-12-21T15:49:39.531 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:39.531 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj666 datafile 2019-12-21T15:49:39.599 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:39.599 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj667 datafile 2019-12-21T15:49:39.645 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:39.645 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj668 datafile 2019-12-21T15:49:39.724 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:39.724 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj669 datafile 2019-12-21T15:49:39.761 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:39.761 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj670 datafile 2019-12-21T15:49:39.833 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:39.833 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj671 datafile 2019-12-21T15:49:39.881 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:39.881 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj672 datafile 2019-12-21T15:49:39.966 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:39.967 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj673 datafile 2019-12-21T15:49:40.042 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:40.042 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj674 datafile 2019-12-21T15:49:40.088 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:40.088 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj675 datafile 2019-12-21T15:49:40.125 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:40.125 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj676 datafile 2019-12-21T15:49:40.170 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:40.170 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj677 datafile 2019-12-21T15:49:40.206 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:40.207 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj678 datafile 2019-12-21T15:49:40.243 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:40.276 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj679 datafile 2019-12-21T15:49:40.326 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:40.326 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj680 datafile 2019-12-21T15:49:40.360 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:40.360 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj681 datafile 2019-12-21T15:49:40.435 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:40.435 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj682 datafile 2019-12-21T15:49:40.535 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:40.536 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj683 datafile 2019-12-21T15:49:40.611 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:40.611 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj684 datafile 2019-12-21T15:49:40.650 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:40.650 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj685 datafile 2019-12-21T15:49:40.685 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:40.685 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj686 datafile 2019-12-21T15:49:40.733 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:40.733 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj687 datafile 2019-12-21T15:49:40.770 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:40.770 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj688 datafile 2019-12-21T15:49:40.861 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:40.861 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj689 datafile 2019-12-21T15:49:40.945 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:40.945 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj690 datafile 2019-12-21T15:49:40.983 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:40.983 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj691 datafile 2019-12-21T15:49:41.087 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:41.087 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj692 datafile 2019-12-21T15:49:41.146 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:41.146 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj693 datafile 2019-12-21T15:49:41.184 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:41.184 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj694 datafile 2019-12-21T15:49:41.220 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:41.220 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj695 datafile 2019-12-21T15:49:41.278 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:41.278 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj696 datafile 2019-12-21T15:49:41.315 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:41.315 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj697 datafile 2019-12-21T15:49:41.350 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:41.350 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj698 datafile 2019-12-21T15:49:41.413 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:41.413 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj699 datafile 2019-12-21T15:49:41.505 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:41.505 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj700 datafile 2019-12-21T15:49:41.543 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:41.544 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj701 datafile 2019-12-21T15:49:41.577 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:41.577 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj702 datafile 2019-12-21T15:49:41.655 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:41.655 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj703 datafile 2019-12-21T15:49:41.694 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:41.694 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj704 datafile 2019-12-21T15:49:41.734 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:41.735 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj705 datafile 2019-12-21T15:49:41.776 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:41.776 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj706 datafile 2019-12-21T15:49:41.811 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:41.812 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj707 datafile 2019-12-21T15:49:41.906 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:41.906 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj708 datafile 2019-12-21T15:49:41.944 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:41.944 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj709 datafile 2019-12-21T15:49:41.995 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:41.995 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj710 datafile 2019-12-21T15:49:42.073 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:42.074 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj711 datafile 2019-12-21T15:49:42.111 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:42.111 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj712 datafile 2019-12-21T15:49:42.145 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:42.145 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj713 datafile 2019-12-21T15:49:42.186 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:42.186 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj714 datafile 2019-12-21T15:49:42.225 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:42.225 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj715 datafile 2019-12-21T15:49:42.270 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:42.270 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj716 datafile 2019-12-21T15:49:42.302 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:42.303 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj717 datafile 2019-12-21T15:49:42.407 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:42.407 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj718 datafile 2019-12-21T15:49:42.482 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:42.483 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj719 datafile 2019-12-21T15:49:42.558 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:42.558 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj720 datafile 2019-12-21T15:49:42.682 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:42.683 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj721 datafile 2019-12-21T15:49:42.733 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:42.734 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj722 datafile 2019-12-21T15:49:42.770 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:42.770 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj723 datafile 2019-12-21T15:49:42.809 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:42.809 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj724 datafile 2019-12-21T15:49:42.851 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:42.851 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj725 datafile 2019-12-21T15:49:42.886 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:42.886 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj726 datafile 2019-12-21T15:49:42.967 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:42.967 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj727 datafile 2019-12-21T15:49:43.012 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:43.012 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj728 datafile 2019-12-21T15:49:43.093 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:43.093 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj729 datafile 2019-12-21T15:49:43.127 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:43.128 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj730 datafile 2019-12-21T15:49:43.167 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:43.168 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj731 datafile 2019-12-21T15:49:43.243 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:43.243 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj732 datafile 2019-12-21T15:49:43.327 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:43.327 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj733 datafile 2019-12-21T15:49:43.427 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:43.428 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj734 datafile 2019-12-21T15:49:43.469 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:43.469 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj735 datafile 2019-12-21T15:49:43.544 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:43.545 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj736 datafile 2019-12-21T15:49:43.586 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:43.586 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj737 datafile 2019-12-21T15:49:43.628 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:43.628 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj738 datafile 2019-12-21T15:49:43.662 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:43.662 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj739 datafile 2019-12-21T15:49:43.710 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:43.710 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj740 datafile 2019-12-21T15:49:43.803 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:43.803 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj741 datafile 2019-12-21T15:49:43.841 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:43.841 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj742 datafile 2019-12-21T15:49:43.886 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:43.886 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj743 datafile 2019-12-21T15:49:43.921 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:43.921 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj744 datafile 2019-12-21T15:49:44.004 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:44.004 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj745 datafile 2019-12-21T15:49:44.180 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:44.180 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj746 datafile 2019-12-21T15:49:44.226 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:44.227 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj747 datafile 2019-12-21T15:49:44.263 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:44.263 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj748 datafile 2019-12-21T15:49:44.322 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:44.322 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj749 datafile 2019-12-21T15:49:44.368 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:44.368 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj750 datafile 2019-12-21T15:49:44.447 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:44.447 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj751 datafile 2019-12-21T15:49:44.493 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:44.493 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj752 datafile 2019-12-21T15:49:44.535 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:44.535 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj753 datafile 2019-12-21T15:49:44.650 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:44.650 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj754 datafile 2019-12-21T15:49:44.806 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:44.806 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj755 datafile 2019-12-21T15:49:45.340 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:45.340 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj756 datafile 2019-12-21T15:49:45.424 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:45.424 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj757 datafile 2019-12-21T15:49:45.468 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:45.469 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj758 datafile 2019-12-21T15:49:45.565 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:45.566 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj759 datafile 2019-12-21T15:49:45.620 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:45.620 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj760 datafile 2019-12-21T15:49:45.661 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:45.661 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj761 datafile 2019-12-21T15:49:45.702 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:45.702 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj762 datafile 2019-12-21T15:49:45.738 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:45.738 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj763 datafile 2019-12-21T15:49:45.816 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:45.816 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj764 datafile 2019-12-21T15:49:45.883 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:45.883 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj765 datafile 2019-12-21T15:49:45.925 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:45.926 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj766 datafile 2019-12-21T15:49:45.960 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:45.960 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj767 datafile 2019-12-21T15:49:46.042 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:46.042 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj768 datafile 2019-12-21T15:49:46.086 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:46.087 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj769 datafile 2019-12-21T15:49:46.167 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:46.167 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj770 datafile 2019-12-21T15:49:46.211 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:46.212 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj771 datafile 2019-12-21T15:49:46.270 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:46.270 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj772 datafile 2019-12-21T15:49:46.325 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:46.326 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj773 datafile 2019-12-21T15:49:46.410 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:46.410 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj774 datafile 2019-12-21T15:49:46.451 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:46.451 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj775 datafile 2019-12-21T15:49:46.526 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:46.527 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj776 datafile 2019-12-21T15:49:46.627 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:46.627 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj777 datafile 2019-12-21T15:49:46.670 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:46.670 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj778 datafile 2019-12-21T15:49:46.704 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:46.705 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj779 datafile 2019-12-21T15:49:46.742 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:46.743 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj780 datafile 2019-12-21T15:49:46.827 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:46.827 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj781 datafile 2019-12-21T15:49:46.870 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:46.870 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj782 datafile 2019-12-21T15:49:46.987 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:46.988 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj783 datafile 2019-12-21T15:49:47.062 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:47.062 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj784 datafile 2019-12-21T15:49:47.100 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:47.101 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj785 datafile 2019-12-21T15:49:47.162 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:47.162 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj786 datafile 2019-12-21T15:49:47.246 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:47.246 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj787 datafile 2019-12-21T15:49:47.286 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:47.286 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj788 datafile 2019-12-21T15:49:47.363 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:47.363 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj789 datafile 2019-12-21T15:49:47.401 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:47.401 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj790 datafile 2019-12-21T15:49:47.444 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:47.444 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj791 datafile 2019-12-21T15:49:47.480 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:47.480 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj792 datafile 2019-12-21T15:49:47.520 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:47.520 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj793 datafile 2019-12-21T15:49:47.613 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:47.614 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj794 datafile 2019-12-21T15:49:47.722 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:47.723 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj795 datafile 2019-12-21T15:49:47.772 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:47.773 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj796 datafile 2019-12-21T15:49:47.819 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:47.820 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj797 datafile 2019-12-21T15:49:47.889 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:47.890 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj798 datafile 2019-12-21T15:49:47.990 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:47.990 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj799 datafile 2019-12-21T15:49:48.040 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:48.040 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj800 datafile 2019-12-21T15:49:48.078 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:48.078 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj801 datafile 2019-12-21T15:49:48.119 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:48.119 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj802 datafile 2019-12-21T15:49:48.207 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:48.207 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj803 datafile 2019-12-21T15:49:48.253 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:48.254 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj804 datafile 2019-12-21T15:49:48.294 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:48.294 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj805 datafile 2019-12-21T15:49:48.330 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:48.330 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj806 datafile 2019-12-21T15:49:48.384 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:48.384 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj807 datafile 2019-12-21T15:49:48.466 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:48.466 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj808 datafile 2019-12-21T15:49:48.541 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:48.541 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj809 datafile 2019-12-21T15:49:48.620 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:48.621 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj810 datafile 2019-12-21T15:49:48.709 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:48.709 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj811 datafile 2019-12-21T15:49:48.753 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:48.754 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj812 datafile 2019-12-21T15:49:48.834 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:48.834 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj813 datafile 2019-12-21T15:49:48.878 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:48.879 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj814 datafile 2019-12-21T15:49:48.920 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:48.920 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj815 datafile 2019-12-21T15:49:48.959 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:48.960 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj816 datafile 2019-12-21T15:49:49.001 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:49.001 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj817 datafile 2019-12-21T15:49:49.044 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:49.044 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj818 datafile 2019-12-21T15:49:49.119 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:49.119 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj819 datafile 2019-12-21T15:49:49.201 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:49.202 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj820 datafile 2019-12-21T15:49:49.310 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:49.311 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj821 datafile 2019-12-21T15:49:49.411 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:49.411 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj822 datafile 2019-12-21T15:49:49.447 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:49.447 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj823 datafile 2019-12-21T15:49:49.485 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:49.486 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj824 datafile 2019-12-21T15:49:49.522 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:49.522 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj825 datafile 2019-12-21T15:49:49.603 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:49.604 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj826 datafile 2019-12-21T15:49:49.687 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:49.687 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj827 datafile 2019-12-21T15:49:49.727 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:49.727 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj828 datafile 2019-12-21T15:49:49.787 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:49.787 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj829 datafile 2019-12-21T15:49:49.871 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:49.871 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj830 datafile 2019-12-21T15:49:49.912 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:49.912 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj831 datafile 2019-12-21T15:49:49.947 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:49.947 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj832 datafile 2019-12-21T15:49:49.980 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:49.980 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj833 datafile 2019-12-21T15:49:50.063 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:50.063 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj834 datafile 2019-12-21T15:49:50.103 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:50.104 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj835 datafile 2019-12-21T15:49:50.188 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:50.189 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj836 datafile 2019-12-21T15:49:50.229 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:50.229 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj837 datafile 2019-12-21T15:49:50.271 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:50.271 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj838 datafile 2019-12-21T15:49:50.322 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:50.322 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj839 datafile 2019-12-21T15:49:50.362 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:50.362 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj840 datafile 2019-12-21T15:49:50.482 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:50.482 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj841 datafile 2019-12-21T15:49:50.556 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:50.556 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj842 datafile 2019-12-21T15:49:50.618 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:50.619 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj843 datafile 2019-12-21T15:49:50.662 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:50.662 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj844 datafile 2019-12-21T15:49:50.840 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:50.841 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj845 datafile 2019-12-21T15:49:50.879 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:50.879 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj846 datafile 2019-12-21T15:49:50.920 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:50.921 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj847 datafile 2019-12-21T15:49:50.955 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:50.955 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj848 datafile 2019-12-21T15:49:51.032 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:51.033 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj849 datafile 2019-12-21T15:49:51.108 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:51.108 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj850 datafile 2019-12-21T15:49:51.216 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:51.217 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj851 datafile 2019-12-21T15:49:51.262 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:51.262 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj852 datafile 2019-12-21T15:49:51.342 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:51.342 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj853 datafile 2019-12-21T15:49:51.379 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:51.379 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj854 datafile 2019-12-21T15:49:51.459 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:51.459 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj855 datafile 2019-12-21T15:49:51.504 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:51.504 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj856 datafile 2019-12-21T15:49:51.546 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:51.546 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj857 datafile 2019-12-21T15:49:51.580 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:51.580 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj858 datafile 2019-12-21T15:49:51.621 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:51.621 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj859 datafile 2019-12-21T15:49:51.718 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:51.718 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj860 datafile 2019-12-21T15:49:51.760 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:51.761 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj861 datafile 2019-12-21T15:49:51.843 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:51.843 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj862 datafile 2019-12-21T15:49:51.910 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:51.911 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj863 datafile 2019-12-21T15:49:51.994 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:51.994 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj864 datafile 2019-12-21T15:49:52.029 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:52.029 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj865 datafile 2019-12-21T15:49:52.069 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:52.069 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj866 datafile 2019-12-21T15:49:52.114 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:52.114 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj867 datafile 2019-12-21T15:49:52.147 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:52.147 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj868 datafile 2019-12-21T15:49:52.219 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:52.219 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj869 datafile 2019-12-21T15:49:52.264 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:52.264 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj870 datafile 2019-12-21T15:49:52.345 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:52.345 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj871 datafile 2019-12-21T15:49:52.389 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:52.389 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj872 datafile 2019-12-21T15:49:52.422 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:52.422 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj873 datafile 2019-12-21T15:49:52.463 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:52.463 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj874 datafile 2019-12-21T15:49:52.502 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:52.502 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj875 datafile 2019-12-21T15:49:52.620 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:52.620 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj876 datafile 2019-12-21T15:49:52.704 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:52.704 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj877 datafile 2019-12-21T15:49:52.747 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:52.747 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj878 datafile 2019-12-21T15:49:52.829 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:52.830 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj879 datafile 2019-12-21T15:49:52.872 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:52.873 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj880 datafile 2019-12-21T15:49:52.910 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:52.911 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj881 datafile 2019-12-21T15:49:53.005 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:53.005 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj882 datafile 2019-12-21T15:49:53.046 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:53.046 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj883 datafile 2019-12-21T15:49:53.088 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:53.088 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj884 datafile 2019-12-21T15:49:53.127 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:53.127 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj885 datafile 2019-12-21T15:49:53.189 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:53.189 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj886 datafile 2019-12-21T15:49:53.323 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:53.323 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj887 datafile 2019-12-21T15:49:53.361 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:53.361 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj888 datafile 2019-12-21T15:49:53.398 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:53.398 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj889 datafile 2019-12-21T15:49:53.431 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:53.431 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj890 datafile 2019-12-21T15:49:53.468 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:53.468 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj891 datafile 2019-12-21T15:49:53.556 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:53.557 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj892 datafile 2019-12-21T15:49:53.632 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:53.632 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj893 datafile 2019-12-21T15:49:53.672 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:53.672 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj894 datafile 2019-12-21T15:49:53.768 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:53.768 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj895 datafile 2019-12-21T15:49:53.824 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:53.824 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj896 datafile 2019-12-21T15:49:53.863 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:53.863 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj897 datafile 2019-12-21T15:49:53.902 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:53.902 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj898 datafile 2019-12-21T15:49:54.008 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:54.008 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj899 datafile 2019-12-21T15:49:54.066 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:54.066 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj900 datafile 2019-12-21T15:49:54.136 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:54.136 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj901 datafile 2019-12-21T15:49:54.217 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:54.217 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj902 datafile 2019-12-21T15:49:54.264 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:54.264 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj903 datafile 2019-12-21T15:49:54.342 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:54.342 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj904 datafile 2019-12-21T15:49:54.434 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:54.435 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj905 datafile 2019-12-21T15:49:54.527 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:54.527 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj906 datafile 2019-12-21T15:49:54.565 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:54.565 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj907 datafile 2019-12-21T15:49:54.661 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:54.661 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj908 datafile 2019-12-21T15:49:54.732 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:54.732 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj909 datafile 2019-12-21T15:49:54.786 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:54.786 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj910 datafile 2019-12-21T15:49:54.823 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:54.823 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj911 datafile 2019-12-21T15:49:54.864 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:54.865 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj912 datafile 2019-12-21T15:49:54.928 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:54.928 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj913 datafile 2019-12-21T15:49:54.973 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:54.973 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj914 datafile 2019-12-21T15:49:55.054 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:55.054 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj915 datafile 2019-12-21T15:49:55.137 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:55.138 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj916 datafile 2019-12-21T15:49:55.221 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:55.221 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj917 datafile 2019-12-21T15:49:55.313 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:55.313 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj918 datafile 2019-12-21T15:49:55.372 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:55.372 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj919 datafile 2019-12-21T15:49:55.422 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:55.422 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj920 datafile 2019-12-21T15:49:55.464 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:55.464 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj921 datafile 2019-12-21T15:49:55.506 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:55.507 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj922 datafile 2019-12-21T15:49:55.544 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:55.544 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj923 datafile 2019-12-21T15:49:55.589 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:55.590 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj924 datafile 2019-12-21T15:49:55.689 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:55.689 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj925 datafile 2019-12-21T15:49:55.731 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:55.731 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj926 datafile 2019-12-21T15:49:55.773 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:55.773 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj927 datafile 2019-12-21T15:49:55.815 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:55.815 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj928 datafile 2019-12-21T15:49:55.956 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:55.956 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj929 datafile 2019-12-21T15:49:56.040 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:56.040 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj930 datafile 2019-12-21T15:49:56.078 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:56.079 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj931 datafile 2019-12-21T15:49:56.131 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:56.132 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj932 datafile 2019-12-21T15:49:56.168 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:56.168 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj933 datafile 2019-12-21T15:49:56.224 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:56.224 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj934 datafile 2019-12-21T15:49:56.316 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:56.316 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj935 datafile 2019-12-21T15:49:56.357 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:56.357 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj936 datafile 2019-12-21T15:49:56.466 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:56.466 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj937 datafile 2019-12-21T15:49:56.504 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:56.504 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj938 datafile 2019-12-21T15:49:56.548 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:56.548 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj939 datafile 2019-12-21T15:49:56.658 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:56.658 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj940 datafile 2019-12-21T15:49:56.698 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:56.699 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj941 datafile 2019-12-21T15:49:56.792 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:56.792 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj942 datafile 2019-12-21T15:49:56.837 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:56.837 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj943 datafile 2019-12-21T15:49:56.880 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:56.880 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj944 datafile 2019-12-21T15:49:56.923 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:56.924 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj945 datafile 2019-12-21T15:49:57.009 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:57.009 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj946 datafile 2019-12-21T15:49:57.046 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:57.047 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj947 datafile 2019-12-21T15:49:57.126 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:57.126 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj948 datafile 2019-12-21T15:49:57.172 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:57.172 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj949 datafile 2019-12-21T15:49:57.269 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:57.269 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj950 datafile 2019-12-21T15:49:57.313 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:57.313 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj951 datafile 2019-12-21T15:49:57.386 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:57.386 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj952 datafile 2019-12-21T15:49:57.428 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:57.429 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj953 datafile 2019-12-21T15:49:57.544 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:57.545 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj954 datafile 2019-12-21T15:49:57.582 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:57.582 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj955 datafile 2019-12-21T15:49:57.622 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:57.622 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj956 datafile 2019-12-21T15:49:57.695 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:57.695 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj957 datafile 2019-12-21T15:49:57.770 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:57.770 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj958 datafile 2019-12-21T15:49:57.846 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:57.846 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj959 datafile 2019-12-21T15:49:57.921 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:57.921 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj960 datafile 2019-12-21T15:49:57.965 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:57.965 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj961 datafile 2019-12-21T15:49:57.999 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:57.999 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj962 datafile 2019-12-21T15:49:58.041 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:58.041 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj963 datafile 2019-12-21T15:49:58.079 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:58.079 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj964 datafile 2019-12-21T15:49:58.122 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:58.122 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj965 datafile 2019-12-21T15:49:58.166 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:58.166 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj966 datafile 2019-12-21T15:49:58.263 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:58.263 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj967 datafile 2019-12-21T15:49:58.339 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:58.339 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj968 datafile 2019-12-21T15:49:58.380 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:58.380 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj969 datafile 2019-12-21T15:49:58.456 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:58.456 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj970 datafile 2019-12-21T15:49:58.491 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:58.491 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj971 datafile 2019-12-21T15:49:58.530 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:58.530 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj972 datafile 2019-12-21T15:49:58.572 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:58.572 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj973 datafile 2019-12-21T15:49:58.690 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:58.690 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj974 datafile 2019-12-21T15:49:58.730 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:58.730 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj975 datafile 2019-12-21T15:49:58.766 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:58.766 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj976 datafile 2019-12-21T15:49:58.849 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:58.849 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj977 datafile 2019-12-21T15:49:58.891 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:58.891 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj978 datafile 2019-12-21T15:49:58.982 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:58.982 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj979 datafile 2019-12-21T15:49:59.019 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:59.020 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj980 datafile 2019-12-21T15:49:59.064 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:59.064 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj981 datafile 2019-12-21T15:49:59.099 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:59.100 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj982 datafile 2019-12-21T15:49:59.159 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:59.159 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj983 datafile 2019-12-21T15:49:59.216 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:59.216 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj984 datafile 2019-12-21T15:49:59.367 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:59.367 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj985 datafile 2019-12-21T15:49:59.451 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:59.451 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj986 datafile 2019-12-21T15:49:59.534 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:59.535 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj987 datafile 2019-12-21T15:49:59.580 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:59.581 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj988 datafile 2019-12-21T15:49:59.685 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:59.685 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj989 datafile 2019-12-21T15:49:59.722 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:59.722 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj990 datafile 2019-12-21T15:49:59.758 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:59.758 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj991 datafile 2019-12-21T15:49:59.847 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:59.848 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj992 datafile 2019-12-21T15:49:59.882 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:59.882 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj993 datafile 2019-12-21T15:49:59.925 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:59.925 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj994 datafile 2019-12-21T15:49:59.964 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:49:59.964 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj995 datafile 2019-12-21T15:50:00.052 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:00.053 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj996 datafile 2019-12-21T15:50:00.136 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:00.136 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj997 datafile 2019-12-21T15:50:00.173 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:00.173 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj998 datafile 2019-12-21T15:50:00.253 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:00.253 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj999 datafile 2019-12-21T15:50:00.292 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:00.292 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test1 put obj1000 datafile 2019-12-21T15:50:00.420 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:72: TEST_recover_unexpected: for i in $(seq 1 $POOLS) 2019-12-21T15:50:00.420 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: seq 1 1000 2019-12-21T15:50:00.422 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:00.422 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj1 datafile 2019-12-21T15:50:00.467 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:00.467 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj2 datafile 2019-12-21T15:50:00.500 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:00.500 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj3 datafile 2019-12-21T15:50:00.536 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:00.536 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj4 datafile 2019-12-21T15:50:00.589 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:00.589 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj5 datafile 2019-12-21T15:50:00.729 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:00.729 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj6 datafile 2019-12-21T15:50:00.766 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:00.766 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj7 datafile 2019-12-21T15:50:00.808 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:00.808 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj8 datafile 2019-12-21T15:50:00.858 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:00.859 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj9 datafile 2019-12-21T15:50:00.930 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:00.930 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj10 datafile 2019-12-21T15:50:00.972 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:00.973 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj11 datafile 2019-12-21T15:50:01.014 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:01.015 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj12 datafile 2019-12-21T15:50:01.058 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:01.059 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj13 datafile 2019-12-21T15:50:01.100 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:01.100 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj14 datafile 2019-12-21T15:50:01.189 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:01.189 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj15 datafile 2019-12-21T15:50:01.233 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:01.234 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj16 datafile 2019-12-21T15:50:01.270 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:01.270 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj17 datafile 2019-12-21T15:50:01.389 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:01.389 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj18 datafile 2019-12-21T15:50:01.431 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:01.431 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj19 datafile 2019-12-21T15:50:01.472 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:01.473 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj20 datafile 2019-12-21T15:50:01.515 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:01.515 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj21 datafile 2019-12-21T15:50:01.556 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:01.556 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj22 datafile 2019-12-21T15:50:01.631 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:01.631 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj23 datafile 2019-12-21T15:50:01.666 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:01.666 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj24 datafile 2019-12-21T15:50:01.773 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:01.773 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj25 datafile 2019-12-21T15:50:01.814 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:01.815 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj26 datafile 2019-12-21T15:50:01.858 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:01.859 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj27 datafile 2019-12-21T15:50:01.940 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:01.940 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj28 datafile 2019-12-21T15:50:01.978 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:01.978 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj29 datafile 2019-12-21T15:50:02.058 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:02.058 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj30 datafile 2019-12-21T15:50:02.096 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:02.096 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj31 datafile 2019-12-21T15:50:02.217 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:02.217 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj32 datafile 2019-12-21T15:50:02.300 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:02.300 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj33 datafile 2019-12-21T15:50:02.350 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:02.351 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj34 datafile 2019-12-21T15:50:02.388 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:02.388 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj35 datafile 2019-12-21T15:50:02.433 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:02.434 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj36 datafile 2019-12-21T15:50:02.481 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:02.481 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj37 datafile 2019-12-21T15:50:02.516 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:02.516 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj38 datafile 2019-12-21T15:50:02.555 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:02.555 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj39 datafile 2019-12-21T15:50:02.704 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:02.704 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj40 datafile 2019-12-21T15:50:02.747 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:02.748 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj41 datafile 2019-12-21T15:50:02.792 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:02.792 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj42 datafile 2019-12-21T15:50:02.871 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:02.871 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj43 datafile 2019-12-21T15:50:02.915 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:02.915 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj44 datafile 2019-12-21T15:50:02.996 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:02.996 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj45 datafile 2019-12-21T15:50:03.046 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:03.046 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj46 datafile 2019-12-21T15:50:03.089 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:03.089 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj47 datafile 2019-12-21T15:50:03.123 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:03.123 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj48 datafile 2019-12-21T15:50:03.181 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:03.182 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj49 datafile 2019-12-21T15:50:03.255 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:03.255 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj50 datafile 2019-12-21T15:50:03.330 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:03.331 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj51 datafile 2019-12-21T15:50:03.373 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:03.373 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj52 datafile 2019-12-21T15:50:03.456 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:03.456 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj53 datafile 2019-12-21T15:50:03.500 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:03.500 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj54 datafile 2019-12-21T15:50:03.539 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:03.539 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj55 datafile 2019-12-21T15:50:03.623 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:03.623 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj56 datafile 2019-12-21T15:50:03.658 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:03.659 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj57 datafile 2019-12-21T15:50:03.765 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:03.765 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj58 datafile 2019-12-21T15:50:03.809 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:03.809 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj59 datafile 2019-12-21T15:50:03.915 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:03.916 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj60 datafile 2019-12-21T15:50:03.956 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:03.956 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj61 datafile 2019-12-21T15:50:03.998 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:03.998 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj62 datafile 2019-12-21T15:50:04.039 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:04.039 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj63 datafile 2019-12-21T15:50:04.124 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:04.124 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj64 datafile 2019-12-21T15:50:04.175 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:04.175 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj65 datafile 2019-12-21T15:50:04.215 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:04.215 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj66 datafile 2019-12-21T15:50:04.316 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:04.316 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj67 datafile 2019-12-21T15:50:04.400 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:04.400 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj68 datafile 2019-12-21T15:50:04.468 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:04.468 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj69 datafile 2019-12-21T15:50:04.506 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:04.506 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj70 datafile 2019-12-21T15:50:04.643 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:04.643 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj71 datafile 2019-12-21T15:50:04.684 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:04.684 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj72 datafile 2019-12-21T15:50:04.721 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:04.721 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj73 datafile 2019-12-21T15:50:04.765 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:04.765 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj74 datafile 2019-12-21T15:50:04.851 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:04.852 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj75 datafile 2019-12-21T15:50:04.892 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:04.892 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj76 datafile 2019-12-21T15:50:04.931 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:04.931 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj77 datafile 2019-12-21T15:50:04.968 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:04.968 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj78 datafile 2019-12-21T15:50:05.043 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:05.043 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj79 datafile 2019-12-21T15:50:05.144 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:05.144 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj80 datafile 2019-12-21T15:50:05.181 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:05.181 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj81 datafile 2019-12-21T15:50:05.236 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:05.236 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj82 datafile 2019-12-21T15:50:05.284 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:05.285 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj83 datafile 2019-12-21T15:50:05.387 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:05.387 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj84 datafile 2019-12-21T15:50:05.423 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:05.423 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj85 datafile 2019-12-21T15:50:05.467 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:05.468 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj86 datafile 2019-12-21T15:50:05.537 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:05.537 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj87 datafile 2019-12-21T15:50:05.582 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:05.582 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj88 datafile 2019-12-21T15:50:05.623 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:05.623 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj89 datafile 2019-12-21T15:50:05.665 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:05.665 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj90 datafile 2019-12-21T15:50:05.710 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:05.710 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj91 datafile 2019-12-21T15:50:05.821 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:05.821 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj92 datafile 2019-12-21T15:50:05.865 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:05.865 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj93 datafile 2019-12-21T15:50:05.901 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:05.901 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj94 datafile 2019-12-21T15:50:06.009 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:06.009 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj95 datafile 2019-12-21T15:50:06.063 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:06.063 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj96 datafile 2019-12-21T15:50:06.122 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:06.123 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj97 datafile 2019-12-21T15:50:06.160 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:06.160 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj98 datafile 2019-12-21T15:50:06.239 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:06.239 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj99 datafile 2019-12-21T15:50:06.277 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:06.277 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj100 datafile 2019-12-21T15:50:06.373 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:06.373 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj101 datafile 2019-12-21T15:50:06.456 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:06.456 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj102 datafile 2019-12-21T15:50:06.501 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:06.501 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj103 datafile 2019-12-21T15:50:06.540 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:06.540 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj104 datafile 2019-12-21T15:50:06.681 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:06.681 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj105 datafile 2019-12-21T15:50:06.765 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:06.765 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj106 datafile 2019-12-21T15:50:06.807 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:06.807 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj107 datafile 2019-12-21T15:50:06.891 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:06.891 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj108 datafile 2019-12-21T15:50:06.929 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:06.929 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj109 datafile 2019-12-21T15:50:07.016 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:07.016 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj110 datafile 2019-12-21T15:50:07.065 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:07.065 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj111 datafile 2019-12-21T15:50:07.110 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:07.110 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj112 datafile 2019-12-21T15:50:07.157 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:07.157 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj113 datafile 2019-12-21T15:50:07.250 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:07.250 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj114 datafile 2019-12-21T15:50:07.293 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:07.294 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj115 datafile 2019-12-21T15:50:07.426 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:07.426 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj116 datafile 2019-12-21T15:50:07.482 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:07.482 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj117 datafile 2019-12-21T15:50:07.532 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:07.532 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj118 datafile 2019-12-21T15:50:07.617 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:07.618 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj119 datafile 2019-12-21T15:50:07.660 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:07.660 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj120 datafile 2019-12-21T15:50:07.699 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:07.699 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj121 datafile 2019-12-21T15:50:07.743 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:07.743 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj122 datafile 2019-12-21T15:50:07.782 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:07.782 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj123 datafile 2019-12-21T15:50:07.859 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:07.859 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj124 datafile 2019-12-21T15:50:07.899 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:07.899 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj125 datafile 2019-12-21T15:50:07.940 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:07.940 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj126 datafile 2019-12-21T15:50:08.035 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:08.035 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj127 datafile 2019-12-21T15:50:08.099 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:08.099 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj128 datafile 2019-12-21T15:50:08.185 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:08.186 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj129 datafile 2019-12-21T15:50:08.232 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:08.232 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj130 datafile 2019-12-21T15:50:08.269 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:08.269 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj131 datafile 2019-12-21T15:50:08.307 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:08.307 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj132 datafile 2019-12-21T15:50:08.369 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:08.369 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj133 datafile 2019-12-21T15:50:08.407 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:08.407 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj134 datafile 2019-12-21T15:50:08.457 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:08.458 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj135 datafile 2019-12-21T15:50:08.545 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:08.545 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj136 datafile 2019-12-21T15:50:08.688 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:08.688 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj137 datafile 2019-12-21T15:50:08.790 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:08.790 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj138 datafile 2019-12-21T15:50:08.846 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:08.846 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj139 datafile 2019-12-21T15:50:08.890 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:08.891 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj140 datafile 2019-12-21T15:50:08.963 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:08.963 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj141 datafile 2019-12-21T15:50:09.008 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:09.008 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj142 datafile 2019-12-21T15:50:09.080 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:09.080 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj143 datafile 2019-12-21T15:50:09.126 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:09.131 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj144 datafile 2019-12-21T15:50:09.264 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:09.264 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj145 datafile 2019-12-21T15:50:09.302 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:09.303 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj146 datafile 2019-12-21T15:50:09.349 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:09.349 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj147 datafile 2019-12-21T15:50:09.440 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:09.440 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj148 datafile 2019-12-21T15:50:09.507 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:09.507 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj149 datafile 2019-12-21T15:50:09.549 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:09.549 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj150 datafile 2019-12-21T15:50:09.585 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:09.585 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj151 datafile 2019-12-21T15:50:09.624 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:09.625 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj152 datafile 2019-12-21T15:50:09.707 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:09.707 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj153 datafile 2019-12-21T15:50:09.745 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:09.745 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj154 datafile 2019-12-21T15:50:09.824 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:09.824 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj155 datafile 2019-12-21T15:50:09.861 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:09.861 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj156 datafile 2019-12-21T15:50:09.958 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:09.958 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj157 datafile 2019-12-21T15:50:09.994 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:09.995 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj158 datafile 2019-12-21T15:50:10.083 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:10.084 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj159 datafile 2019-12-21T15:50:10.124 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:10.125 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj160 datafile 2019-12-21T15:50:10.161 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:10.161 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj161 datafile 2019-12-21T15:50:10.208 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:10.208 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj162 datafile 2019-12-21T15:50:10.257 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:10.257 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj163 datafile 2019-12-21T15:50:10.353 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:10.353 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj164 datafile 2019-12-21T15:50:10.400 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:10.400 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj165 datafile 2019-12-21T15:50:10.436 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:10.436 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj166 datafile 2019-12-21T15:50:10.474 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:10.475 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj167 datafile 2019-12-21T15:50:10.559 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:10.560 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj168 datafile 2019-12-21T15:50:10.685 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:10.685 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj169 datafile 2019-12-21T15:50:10.794 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:10.794 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj170 datafile 2019-12-21T15:50:10.832 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:10.832 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj171 datafile 2019-12-21T15:50:10.882 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:10.883 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj172 datafile 2019-12-21T15:50:10.941 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:10.941 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj173 datafile 2019-12-21T15:50:11.035 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:11.036 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj174 datafile 2019-12-21T15:50:11.082 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:11.083 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj175 datafile 2019-12-21T15:50:11.133 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:11.133 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj176 datafile 2019-12-21T15:50:11.261 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:11.262 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj177 datafile 2019-12-21T15:50:11.308 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:11.308 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj178 datafile 2019-12-21T15:50:11.387 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:11.387 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj179 datafile 2019-12-21T15:50:11.433 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:11.433 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj180 datafile 2019-12-21T15:50:11.520 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:11.521 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj181 datafile 2019-12-21T15:50:11.574 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:11.574 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj182 datafile 2019-12-21T15:50:11.624 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:11.624 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj183 datafile 2019-12-21T15:50:11.704 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:11.704 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj184 datafile 2019-12-21T15:50:11.797 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:11.797 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj185 datafile 2019-12-21T15:50:11.863 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:11.863 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj186 datafile 2019-12-21T15:50:11.908 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:11.908 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj187 datafile 2019-12-21T15:50:11.957 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:11.958 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj188 datafile 2019-12-21T15:50:11.999 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:12.000 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj189 datafile 2019-12-21T15:50:12.041 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:12.041 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj190 datafile 2019-12-21T15:50:12.139 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:12.139 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj191 datafile 2019-12-21T15:50:12.214 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:12.214 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj192 datafile 2019-12-21T15:50:12.258 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:12.258 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj193 datafile 2019-12-21T15:50:12.326 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:12.327 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj194 datafile 2019-12-21T15:50:12.406 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:12.407 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj195 datafile 2019-12-21T15:50:12.507 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:12.507 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj196 datafile 2019-12-21T15:50:12.560 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:12.560 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj197 datafile 2019-12-21T15:50:12.616 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:12.617 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj198 datafile 2019-12-21T15:50:12.659 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:12.660 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj199 datafile 2019-12-21T15:50:12.700 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:12.700 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj200 datafile 2019-12-21T15:50:12.790 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:12.791 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj201 datafile 2019-12-21T15:50:12.833 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:12.833 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj202 datafile 2019-12-21T15:50:12.916 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:12.916 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj203 datafile 2019-12-21T15:50:12.953 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:12.954 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj204 datafile 2019-12-21T15:50:13.017 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:13.017 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj205 datafile 2019-12-21T15:50:13.054 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:13.054 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj206 datafile 2019-12-21T15:50:13.133 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:13.133 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj207 datafile 2019-12-21T15:50:13.237 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:13.237 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj208 datafile 2019-12-21T15:50:13.275 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:13.275 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj209 datafile 2019-12-21T15:50:13.317 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:13.317 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj210 datafile 2019-12-21T15:50:13.358 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:13.358 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj211 datafile 2019-12-21T15:50:13.434 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:13.434 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj212 datafile 2019-12-21T15:50:13.476 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:13.476 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj213 datafile 2019-12-21T15:50:13.517 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:13.517 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj214 datafile 2019-12-21T15:50:13.584 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:13.585 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj215 datafile 2019-12-21T15:50:13.712 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:13.712 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj216 datafile 2019-12-21T15:50:13.767 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:13.767 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj217 datafile 2019-12-21T15:50:13.808 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:13.809 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj218 datafile 2019-12-21T15:50:13.902 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:13.902 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj219 datafile 2019-12-21T15:50:13.950 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:13.950 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj220 datafile 2019-12-21T15:50:14.036 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:14.036 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj221 datafile 2019-12-21T15:50:14.075 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:14.075 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj222 datafile 2019-12-21T15:50:14.125 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:14.125 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj223 datafile 2019-12-21T15:50:14.159 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:14.159 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj224 datafile 2019-12-21T15:50:14.237 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:14.237 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj225 datafile 2019-12-21T15:50:14.284 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:14.284 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj226 datafile 2019-12-21T15:50:14.334 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:14.334 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj227 datafile 2019-12-21T15:50:14.384 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:14.384 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj228 datafile 2019-12-21T15:50:14.470 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:14.470 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj229 datafile 2019-12-21T15:50:14.545 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:14.545 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj230 datafile 2019-12-21T15:50:14.663 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:14.663 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj231 datafile 2019-12-21T15:50:14.708 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:14.709 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj232 datafile 2019-12-21T15:50:14.750 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:14.750 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj233 datafile 2019-12-21T15:50:14.871 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:14.872 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj234 datafile 2019-12-21T15:50:14.909 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:14.909 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj235 datafile 2019-12-21T15:50:14.950 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:14.950 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj236 datafile 2019-12-21T15:50:14.992 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:14.992 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj237 datafile 2019-12-21T15:50:15.055 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:15.055 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj238 datafile 2019-12-21T15:50:15.093 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:15.093 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj239 datafile 2019-12-21T15:50:15.150 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:15.150 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj240 datafile 2019-12-21T15:50:15.239 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:15.239 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj241 datafile 2019-12-21T15:50:15.284 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:15.284 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj242 datafile 2019-12-21T15:50:15.372 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:15.373 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj243 datafile 2019-12-21T15:50:15.464 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:15.465 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj244 datafile 2019-12-21T15:50:15.557 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:15.557 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj245 datafile 2019-12-21T15:50:15.609 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:15.609 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj246 datafile 2019-12-21T15:50:15.659 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:15.659 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj247 datafile 2019-12-21T15:50:15.709 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:15.709 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj248 datafile 2019-12-21T15:50:15.799 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:15.799 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj249 datafile 2019-12-21T15:50:15.842 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:15.842 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj250 datafile 2019-12-21T15:50:15.883 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:15.883 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj251 datafile 2019-12-21T15:50:15.957 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:15.958 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj252 datafile 2019-12-21T15:50:16.009 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:16.009 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj253 datafile 2019-12-21T15:50:16.051 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:16.052 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj254 datafile 2019-12-21T15:50:16.093 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:16.093 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj255 datafile 2019-12-21T15:50:16.200 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:16.200 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj256 datafile 2019-12-21T15:50:16.283 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:16.284 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj257 datafile 2019-12-21T15:50:16.440 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:16.440 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj258 datafile 2019-12-21T15:50:16.563 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:16.563 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj259 datafile 2019-12-21T15:50:17.009 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:17.010 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj260 datafile 2019-12-21T15:50:17.051 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:17.051 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj261 datafile 2019-12-21T15:50:17.160 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:17.160 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj262 datafile 2019-12-21T15:50:17.200 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:17.201 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj263 datafile 2019-12-21T15:50:17.268 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:17.268 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj264 datafile 2019-12-21T15:50:17.360 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:17.361 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj265 datafile 2019-12-21T15:50:17.444 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:17.444 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj266 datafile 2019-12-21T15:50:17.486 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:17.487 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj267 datafile 2019-12-21T15:50:17.570 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:17.570 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj268 datafile 2019-12-21T15:50:17.609 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:17.609 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj269 datafile 2019-12-21T15:50:17.651 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:17.651 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj270 datafile 2019-12-21T15:50:17.692 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:17.692 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj271 datafile 2019-12-21T15:50:17.734 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:17.735 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj272 datafile 2019-12-21T15:50:17.828 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:17.829 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj273 datafile 2019-12-21T15:50:17.912 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:17.913 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj274 datafile 2019-12-21T15:50:17.951 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:17.951 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj275 datafile 2019-12-21T15:50:18.018 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:18.018 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj276 datafile 2019-12-21T15:50:18.096 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:18.096 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj277 datafile 2019-12-21T15:50:18.171 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:18.171 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj278 datafile 2019-12-21T15:50:18.218 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:18.218 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj279 datafile 2019-12-21T15:50:18.296 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:18.296 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj280 datafile 2019-12-21T15:50:18.380 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:18.380 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj281 datafile 2019-12-21T15:50:18.426 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:18.426 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj282 datafile 2019-12-21T15:50:18.467 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:18.467 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj283 datafile 2019-12-21T15:50:18.509 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:18.510 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj284 datafile 2019-12-21T15:50:18.551 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:18.551 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj285 datafile 2019-12-21T15:50:18.622 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:18.623 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj286 datafile 2019-12-21T15:50:18.668 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:18.668 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj287 datafile 2019-12-21T15:50:18.764 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:18.765 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj288 datafile 2019-12-21T15:50:18.809 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:18.809 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj289 datafile 2019-12-21T15:50:18.851 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:18.851 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj290 datafile 2019-12-21T15:50:18.931 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:18.932 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj291 datafile 2019-12-21T15:50:19.032 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:19.032 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj292 datafile 2019-12-21T15:50:19.076 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:19.076 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj293 datafile 2019-12-21T15:50:19.149 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:19.149 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj294 datafile 2019-12-21T15:50:19.193 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:19.193 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj295 datafile 2019-12-21T15:50:19.259 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:19.259 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj296 datafile 2019-12-21T15:50:19.375 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:19.375 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj297 datafile 2019-12-21T15:50:19.427 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:19.427 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj298 datafile 2019-12-21T15:50:19.477 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:19.477 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj299 datafile 2019-12-21T15:50:19.576 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:19.576 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj300 datafile 2019-12-21T15:50:19.618 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:19.618 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj301 datafile 2019-12-21T15:50:19.659 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:19.660 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj302 datafile 2019-12-21T15:50:19.700 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:19.700 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj303 datafile 2019-12-21T15:50:19.776 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:19.776 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj304 datafile 2019-12-21T15:50:19.817 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:19.817 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj305 datafile 2019-12-21T15:50:19.860 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:19.860 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj306 datafile 2019-12-21T15:50:19.900 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:19.901 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj307 datafile 2019-12-21T15:50:19.951 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:19.951 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj308 datafile 2019-12-21T15:50:20.001 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:20.001 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj309 datafile 2019-12-21T15:50:20.085 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:20.085 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj310 datafile 2019-12-21T15:50:20.195 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:20.195 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj311 datafile 2019-12-21T15:50:20.269 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:20.269 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj312 datafile 2019-12-21T15:50:20.349 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:20.349 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj313 datafile 2019-12-21T15:50:20.437 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:20.437 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj314 datafile 2019-12-21T15:50:20.476 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:20.476 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj315 datafile 2019-12-21T15:50:20.518 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:20.518 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj316 datafile 2019-12-21T15:50:20.560 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:20.561 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj317 datafile 2019-12-21T15:50:20.637 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:20.637 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj318 datafile 2019-12-21T15:50:20.676 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:20.676 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj319 datafile 2019-12-21T15:50:20.718 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:20.718 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj320 datafile 2019-12-21T15:50:20.856 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:20.856 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj321 datafile 2019-12-21T15:50:20.938 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:20.938 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj322 datafile 2019-12-21T15:50:21.030 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:21.030 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj323 datafile 2019-12-21T15:50:21.105 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:21.105 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj324 datafile 2019-12-21T15:50:21.144 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:21.144 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj325 datafile 2019-12-21T15:50:21.222 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:21.223 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj326 datafile 2019-12-21T15:50:21.268 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:21.268 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj327 datafile 2019-12-21T15:50:21.310 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:21.310 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj328 datafile 2019-12-21T15:50:21.351 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:21.352 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj329 datafile 2019-12-21T15:50:21.431 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:21.431 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj330 datafile 2019-12-21T15:50:21.469 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:21.469 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj331 datafile 2019-12-21T15:50:21.542 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:21.543 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj332 datafile 2019-12-21T15:50:21.592 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:21.592 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj333 datafile 2019-12-21T15:50:21.634 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:21.634 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj334 datafile 2019-12-21T15:50:21.723 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:21.723 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj335 datafile 2019-12-21T15:50:21.790 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:21.790 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj336 datafile 2019-12-21T15:50:21.835 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:21.835 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj337 datafile 2019-12-21T15:50:21.876 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:21.876 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj338 datafile 2019-12-21T15:50:21.991 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:21.991 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj339 datafile 2019-12-21T15:50:22.074 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:22.074 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj340 datafile 2019-12-21T15:50:22.118 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:22.119 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj341 datafile 2019-12-21T15:50:22.161 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:22.161 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj342 datafile 2019-12-21T15:50:22.258 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:22.259 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj343 datafile 2019-12-21T15:50:22.294 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:22.294 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj344 datafile 2019-12-21T15:50:22.375 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:22.375 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj345 datafile 2019-12-21T15:50:22.451 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:22.451 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj346 datafile 2019-12-21T15:50:22.490 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:22.490 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj347 datafile 2019-12-21T15:50:22.534 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:22.535 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj348 datafile 2019-12-21T15:50:22.651 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:22.651 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj349 datafile 2019-12-21T15:50:22.703 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:22.703 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj350 datafile 2019-12-21T15:50:22.736 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:22.736 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj351 datafile 2019-12-21T15:50:22.776 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:22.776 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj352 datafile 2019-12-21T15:50:22.852 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:22.852 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj353 datafile 2019-12-21T15:50:22.961 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:22.961 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj354 datafile 2019-12-21T15:50:23.044 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:23.045 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj355 datafile 2019-12-21T15:50:23.085 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:23.085 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj356 datafile 2019-12-21T15:50:23.161 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:23.161 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj357 datafile 2019-12-21T15:50:23.211 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:23.211 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj358 datafile 2019-12-21T15:50:23.295 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:23.295 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj359 datafile 2019-12-21T15:50:23.345 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:23.345 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj360 datafile 2019-12-21T15:50:23.395 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:23.395 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj361 datafile 2019-12-21T15:50:23.479 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:23.479 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj362 datafile 2019-12-21T15:50:23.518 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:23.518 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj363 datafile 2019-12-21T15:50:23.560 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:23.560 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj364 datafile 2019-12-21T15:50:23.603 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:23.603 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj365 datafile 2019-12-21T15:50:23.651 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:23.651 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj366 datafile 2019-12-21T15:50:23.763 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:23.763 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj367 datafile 2019-12-21T15:50:23.810 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:23.810 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj368 datafile 2019-12-21T15:50:23.863 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:23.863 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj369 datafile 2019-12-21T15:50:23.901 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:23.901 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj370 datafile 2019-12-21T15:50:23.944 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:23.944 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj371 datafile 2019-12-21T15:50:24.022 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:24.022 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj372 datafile 2019-12-21T15:50:24.072 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:24.072 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj373 datafile 2019-12-21T15:50:24.117 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:24.117 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj374 datafile 2019-12-21T15:50:24.197 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:24.197 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj375 datafile 2019-12-21T15:50:24.306 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:24.307 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj376 datafile 2019-12-21T15:50:24.390 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:24.390 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj377 datafile 2019-12-21T15:50:24.465 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:24.465 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj378 datafile 2019-12-21T15:50:24.509 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:24.510 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj379 datafile 2019-12-21T15:50:24.553 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:24.553 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj380 datafile 2019-12-21T15:50:24.634 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:24.634 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj381 datafile 2019-12-21T15:50:24.677 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:24.677 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj382 datafile 2019-12-21T15:50:24.718 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:24.718 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj383 datafile 2019-12-21T15:50:24.760 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:24.761 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj384 datafile 2019-12-21T15:50:24.807 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:24.807 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj385 datafile 2019-12-21T15:50:24.883 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:24.883 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj386 datafile 2019-12-21T15:50:24.927 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:24.927 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj387 datafile 2019-12-21T15:50:25.049 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:25.049 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj388 datafile 2019-12-21T15:50:25.150 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:25.151 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj389 datafile 2019-12-21T15:50:25.192 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:25.192 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj390 datafile 2019-12-21T15:50:25.268 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:25.268 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj391 datafile 2019-12-21T15:50:25.310 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:25.311 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj392 datafile 2019-12-21T15:50:25.352 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:25.352 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj393 datafile 2019-12-21T15:50:25.386 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:25.387 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj394 datafile 2019-12-21T15:50:25.428 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:25.429 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj395 datafile 2019-12-21T15:50:25.469 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:25.469 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj396 datafile 2019-12-21T15:50:25.526 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:25.526 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj397 datafile 2019-12-21T15:50:25.618 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:25.618 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj398 datafile 2019-12-21T15:50:25.702 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:25.702 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj399 datafile 2019-12-21T15:50:25.803 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:25.803 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj400 datafile 2019-12-21T15:50:25.878 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:25.878 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj401 datafile 2019-12-21T15:50:25.962 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:25.962 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj402 datafile 2019-12-21T15:50:26.004 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:26.004 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj403 datafile 2019-12-21T15:50:26.044 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:26.044 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj404 datafile 2019-12-21T15:50:26.087 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:26.087 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj405 datafile 2019-12-21T15:50:26.170 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:26.171 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj406 datafile 2019-12-21T15:50:26.237 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:26.238 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj407 datafile 2019-12-21T15:50:26.276 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:26.276 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj408 datafile 2019-12-21T15:50:26.319 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:26.319 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj409 datafile 2019-12-21T15:50:26.404 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:26.404 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj410 datafile 2019-12-21T15:50:26.505 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:26.505 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj411 datafile 2019-12-21T15:50:26.552 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:26.552 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj412 datafile 2019-12-21T15:50:26.672 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:26.672 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj413 datafile 2019-12-21T15:50:26.712 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:26.713 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj414 datafile 2019-12-21T15:50:26.751 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:26.751 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj415 datafile 2019-12-21T15:50:26.795 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:26.795 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj416 datafile 2019-12-21T15:50:26.847 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:26.847 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj417 datafile 2019-12-21T15:50:26.931 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:26.931 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj418 datafile 2019-12-21T15:50:26.977 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:26.977 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj419 datafile 2019-12-21T15:50:27.019 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:27.020 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj420 datafile 2019-12-21T15:50:27.215 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:27.215 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj421 datafile 2019-12-21T15:50:27.300 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:27.300 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj422 datafile 2019-12-21T15:50:27.344 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:27.345 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj423 datafile 2019-12-21T15:50:27.394 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:27.395 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj424 datafile 2019-12-21T15:50:27.482 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:27.483 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj425 datafile 2019-12-21T15:50:27.526 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:27.526 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj426 datafile 2019-12-21T15:50:27.570 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:27.570 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj427 datafile 2019-12-21T15:50:27.610 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:27.610 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj428 datafile 2019-12-21T15:50:27.652 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:27.652 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj429 datafile 2019-12-21T15:50:27.687 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:27.688 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj430 datafile 2019-12-21T15:50:27.728 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:27.728 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj431 datafile 2019-12-21T15:50:27.800 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:27.800 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj432 datafile 2019-12-21T15:50:27.868 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:27.868 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj433 datafile 2019-12-21T15:50:27.952 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:27.952 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj434 datafile 2019-12-21T15:50:28.025 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:28.026 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj435 datafile 2019-12-21T15:50:28.068 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:28.068 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj436 datafile 2019-12-21T15:50:28.111 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:28.112 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj437 datafile 2019-12-21T15:50:28.151 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:28.152 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj438 datafile 2019-12-21T15:50:28.187 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:28.188 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj439 datafile 2019-12-21T15:50:28.276 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:28.277 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj440 datafile 2019-12-21T15:50:28.320 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:28.320 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj441 datafile 2019-12-21T15:50:28.393 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:28.394 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj442 datafile 2019-12-21T15:50:28.429 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:28.429 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj443 datafile 2019-12-21T15:50:28.470 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:28.470 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj444 datafile 2019-12-21T15:50:28.511 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:28.512 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj445 datafile 2019-12-21T15:50:28.580 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:28.580 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj446 datafile 2019-12-21T15:50:28.644 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:28.644 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj447 datafile 2019-12-21T15:50:28.679 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:28.680 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj448 datafile 2019-12-21T15:50:28.720 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:28.720 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj449 datafile 2019-12-21T15:50:28.828 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:28.828 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj450 datafile 2019-12-21T15:50:28.868 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:28.868 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj451 datafile 2019-12-21T15:50:28.904 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:28.905 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj452 datafile 2019-12-21T15:50:28.986 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:28.987 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj453 datafile 2019-12-21T15:50:29.070 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:29.070 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj454 datafile 2019-12-21T15:50:29.154 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:29.154 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj455 datafile 2019-12-21T15:50:29.195 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:29.195 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj456 datafile 2019-12-21T15:50:29.305 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:29.305 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj457 datafile 2019-12-21T15:50:29.413 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:29.414 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj458 datafile 2019-12-21T15:50:29.514 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:29.514 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj459 datafile 2019-12-21T15:50:29.554 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:29.555 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj460 datafile 2019-12-21T15:50:29.631 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:29.631 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj461 datafile 2019-12-21T15:50:29.671 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:29.671 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj462 datafile 2019-12-21T15:50:29.721 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:29.721 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj463 datafile 2019-12-21T15:50:29.763 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:29.763 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj464 datafile 2019-12-21T15:50:29.848 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:29.848 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj465 datafile 2019-12-21T15:50:29.894 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:29.894 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj466 datafile 2019-12-21T15:50:29.936 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:29.936 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj467 datafile 2019-12-21T15:50:29.979 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:29.979 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj468 datafile 2019-12-21T15:50:30.021 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:30.021 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj469 datafile 2019-12-21T15:50:30.063 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:30.063 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj470 datafile 2019-12-21T15:50:30.174 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:30.174 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj471 datafile 2019-12-21T15:50:30.249 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:30.249 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj472 datafile 2019-12-21T15:50:30.293 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:30.293 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj473 datafile 2019-12-21T15:50:30.335 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:30.335 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj474 datafile 2019-12-21T15:50:30.380 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:30.380 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj475 datafile 2019-12-21T15:50:30.457 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:30.458 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj476 datafile 2019-12-21T15:50:30.494 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:30.494 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj477 datafile 2019-12-21T15:50:30.592 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:30.592 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj478 datafile 2019-12-21T15:50:30.725 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:30.726 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj479 datafile 2019-12-21T15:50:30.769 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:30.769 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj480 datafile 2019-12-21T15:50:30.822 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:30.822 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj481 datafile 2019-12-21T15:50:30.872 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:30.872 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj482 datafile 2019-12-21T15:50:30.934 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:30.934 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj483 datafile 2019-12-21T15:50:30.985 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:30.985 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj484 datafile 2019-12-21T15:50:31.027 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:31.027 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj485 datafile 2019-12-21T15:50:31.109 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:31.109 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj486 datafile 2019-12-21T15:50:31.152 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:31.153 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj487 datafile 2019-12-21T15:50:31.235 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:31.235 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj488 datafile 2019-12-21T15:50:31.319 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:31.319 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj489 datafile 2019-12-21T15:50:31.360 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:31.360 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj490 datafile 2019-12-21T15:50:31.435 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:31.435 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj491 datafile 2019-12-21T15:50:31.519 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:31.519 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj492 datafile 2019-12-21T15:50:31.611 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:31.611 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj493 datafile 2019-12-21T15:50:31.660 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:31.660 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj494 datafile 2019-12-21T15:50:31.745 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:31.745 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj495 datafile 2019-12-21T15:50:31.789 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:31.789 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj496 datafile 2019-12-21T15:50:31.835 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:31.835 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj497 datafile 2019-12-21T15:50:31.885 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:31.885 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj498 datafile 2019-12-21T15:50:31.930 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:31.930 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj499 datafile 2019-12-21T15:50:31.969 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:31.969 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj500 datafile 2019-12-21T15:50:32.014 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:32.014 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj501 datafile 2019-12-21T15:50:32.120 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:32.120 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj502 datafile 2019-12-21T15:50:32.229 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:32.229 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj503 datafile 2019-12-21T15:50:32.313 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:32.313 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj504 datafile 2019-12-21T15:50:32.405 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:32.405 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj505 datafile 2019-12-21T15:50:32.447 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:32.447 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj506 datafile 2019-12-21T15:50:32.485 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:32.486 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj507 datafile 2019-12-21T15:50:32.530 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:32.531 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj508 datafile 2019-12-21T15:50:32.630 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:32.630 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj509 datafile 2019-12-21T15:50:32.705 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:32.706 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj510 datafile 2019-12-21T15:50:32.823 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:32.823 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj511 datafile 2019-12-21T15:50:32.873 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:32.873 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj512 datafile 2019-12-21T15:50:32.956 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:32.957 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj513 datafile 2019-12-21T15:50:33.002 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:33.002 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj514 datafile 2019-12-21T15:50:33.047 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:33.047 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj515 datafile 2019-12-21T15:50:33.085 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:33.085 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj516 datafile 2019-12-21T15:50:33.199 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:33.199 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj517 datafile 2019-12-21T15:50:33.243 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:33.244 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj518 datafile 2019-12-21T15:50:33.289 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:33.289 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj519 datafile 2019-12-21T15:50:33.327 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:33.327 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj520 datafile 2019-12-21T15:50:33.369 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:33.369 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj521 datafile 2019-12-21T15:50:33.414 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:33.414 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj522 datafile 2019-12-21T15:50:33.452 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:33.452 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj523 datafile 2019-12-21T15:50:33.506 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:33.506 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj524 datafile 2019-12-21T15:50:33.608 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:33.608 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj525 datafile 2019-12-21T15:50:33.646 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:33.646 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj526 datafile 2019-12-21T15:50:33.689 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:33.689 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj527 datafile 2019-12-21T15:50:33.727 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:33.728 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj528 datafile 2019-12-21T15:50:33.808 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:33.808 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj529 datafile 2019-12-21T15:50:33.900 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:33.900 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj530 datafile 2019-12-21T15:50:33.938 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:33.938 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj531 datafile 2019-12-21T15:50:34.026 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:34.026 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj532 datafile 2019-12-21T15:50:34.078 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:34.078 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj533 datafile 2019-12-21T15:50:34.151 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:34.151 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj534 datafile 2019-12-21T15:50:34.215 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:34.215 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj535 datafile 2019-12-21T15:50:34.360 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:34.361 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj536 datafile 2019-12-21T15:50:34.402 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:34.402 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj537 datafile 2019-12-21T15:50:34.448 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:34.448 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj538 datafile 2019-12-21T15:50:34.486 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:34.486 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj539 datafile 2019-12-21T15:50:34.552 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:34.553 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj540 datafile 2019-12-21T15:50:34.636 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:34.636 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj541 datafile 2019-12-21T15:50:34.721 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:34.721 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj542 datafile 2019-12-21T15:50:34.795 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:34.795 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj543 datafile 2019-12-21T15:50:34.903 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:34.903 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj544 datafile 2019-12-21T15:50:34.952 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:34.952 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj545 datafile 2019-12-21T15:50:34.998 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:34.998 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj546 datafile 2019-12-21T15:50:35.035 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:35.036 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj547 datafile 2019-12-21T15:50:35.121 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:35.121 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj548 datafile 2019-12-21T15:50:35.204 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:35.204 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj549 datafile 2019-12-21T15:50:35.245 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:35.245 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj550 datafile 2019-12-21T15:50:35.290 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:35.290 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj551 datafile 2019-12-21T15:50:35.397 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:35.397 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj552 datafile 2019-12-21T15:50:35.439 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:35.440 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj553 datafile 2019-12-21T15:50:35.478 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:35.478 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj554 datafile 2019-12-21T15:50:35.547 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:35.547 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj555 datafile 2019-12-21T15:50:35.615 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:35.615 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj556 datafile 2019-12-21T15:50:35.697 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:35.697 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj557 datafile 2019-12-21T15:50:35.740 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:35.740 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj558 datafile 2019-12-21T15:50:35.823 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:35.823 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj559 datafile 2019-12-21T15:50:35.878 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:35.878 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj560 datafile 2019-12-21T15:50:35.920 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:35.920 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj561 datafile 2019-12-21T15:50:36.023 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:36.023 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj562 datafile 2019-12-21T15:50:36.078 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:36.078 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj563 datafile 2019-12-21T15:50:36.140 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:36.140 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj564 datafile 2019-12-21T15:50:36.187 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:36.187 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj565 datafile 2019-12-21T15:50:36.227 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:36.228 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj566 datafile 2019-12-21T15:50:36.270 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:36.270 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj567 datafile 2019-12-21T15:50:36.366 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:36.366 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj568 datafile 2019-12-21T15:50:36.450 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:36.450 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj569 datafile 2019-12-21T15:50:36.494 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:36.495 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj570 datafile 2019-12-21T15:50:36.536 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:36.537 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj571 datafile 2019-12-21T15:50:36.650 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:36.650 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj572 datafile 2019-12-21T15:50:36.695 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:36.695 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj573 datafile 2019-12-21T15:50:36.740 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:36.740 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj574 datafile 2019-12-21T15:50:36.817 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:36.817 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj575 datafile 2019-12-21T15:50:36.918 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:36.918 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj576 datafile 2019-12-21T15:50:36.955 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:36.955 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj577 datafile 2019-12-21T15:50:37.020 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:37.020 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj578 datafile 2019-12-21T15:50:37.110 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:37.110 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj579 datafile 2019-12-21T15:50:37.161 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:37.161 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj580 datafile 2019-12-21T15:50:37.243 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:37.243 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj581 datafile 2019-12-21T15:50:37.295 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:37.295 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj582 datafile 2019-12-21T15:50:37.394 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:37.394 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj583 datafile 2019-12-21T15:50:37.437 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:37.437 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj584 datafile 2019-12-21T15:50:37.482 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:37.482 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj585 datafile 2019-12-21T15:50:37.519 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:37.519 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj586 datafile 2019-12-21T15:50:37.577 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:37.577 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj587 datafile 2019-12-21T15:50:37.611 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:37.612 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj588 datafile 2019-12-21T15:50:37.723 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:37.723 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj589 datafile 2019-12-21T15:50:37.764 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:37.764 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj590 datafile 2019-12-21T15:50:37.845 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:37.845 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj591 datafile 2019-12-21T15:50:37.920 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:37.920 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj592 datafile 2019-12-21T15:50:38.004 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:38.004 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj593 datafile 2019-12-21T15:50:38.087 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:38.088 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj594 datafile 2019-12-21T15:50:38.132 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:38.132 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj595 datafile 2019-12-21T15:50:38.229 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:38.230 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj596 datafile 2019-12-21T15:50:38.274 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:38.274 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj597 datafile 2019-12-21T15:50:38.312 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:38.312 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj598 datafile 2019-12-21T15:50:38.353 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:38.354 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj599 datafile 2019-12-21T15:50:38.455 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:38.455 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj600 datafile 2019-12-21T15:50:38.539 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:38.539 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj601 datafile 2019-12-21T15:50:38.578 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:38.579 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj602 datafile 2019-12-21T15:50:38.701 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:38.701 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj603 datafile 2019-12-21T15:50:38.781 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:38.781 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj604 datafile 2019-12-21T15:50:38.824 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:38.824 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj605 datafile 2019-12-21T15:50:38.882 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:38.882 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj606 datafile 2019-12-21T15:50:38.924 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:38.924 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj607 datafile 2019-12-21T15:50:38.962 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:38.962 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj608 datafile 2019-12-21T15:50:39.004 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:39.004 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj609 datafile 2019-12-21T15:50:39.091 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:39.091 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj610 datafile 2019-12-21T15:50:39.182 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:39.182 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj611 datafile 2019-12-21T15:50:39.228 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:39.229 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj612 datafile 2019-12-21T15:50:39.287 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:39.287 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj613 datafile 2019-12-21T15:50:39.354 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:39.354 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj614 datafile 2019-12-21T15:50:39.391 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:39.391 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj615 datafile 2019-12-21T15:50:39.466 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:39.467 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj616 datafile 2019-12-21T15:50:39.512 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:39.512 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj617 datafile 2019-12-21T15:50:39.554 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:39.554 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj618 datafile 2019-12-21T15:50:39.625 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:39.626 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj619 datafile 2019-12-21T15:50:39.709 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:39.709 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj620 datafile 2019-12-21T15:50:39.801 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:39.838 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj621 datafile 2019-12-21T15:50:39.868 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:39.868 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj622 datafile 2019-12-21T15:50:39.912 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:39.912 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj623 datafile 2019-12-21T15:50:39.994 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:39.994 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj624 datafile 2019-12-21T15:50:40.031 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:40.031 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj625 datafile 2019-12-21T15:50:40.066 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:40.066 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj626 datafile 2019-12-21T15:50:40.161 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:40.161 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj627 datafile 2019-12-21T15:50:40.244 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:40.245 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj628 datafile 2019-12-21T15:50:40.296 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:40.296 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj629 datafile 2019-12-21T15:50:40.337 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:40.337 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj630 datafile 2019-12-21T15:50:40.379 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:40.379 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj631 datafile 2019-12-21T15:50:40.414 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:40.414 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj632 datafile 2019-12-21T15:50:40.449 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:40.450 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj633 datafile 2019-12-21T15:50:40.591 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:40.591 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj634 datafile 2019-12-21T15:50:40.637 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:40.638 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj635 datafile 2019-12-21T15:50:40.673 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:40.673 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj636 datafile 2019-12-21T15:50:40.708 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:40.708 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj637 datafile 2019-12-21T15:50:40.745 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:40.745 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj638 datafile 2019-12-21T15:50:40.787 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:40.787 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj639 datafile 2019-12-21T15:50:40.829 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:40.829 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj640 datafile 2019-12-21T15:50:40.870 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:40.870 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj641 datafile 2019-12-21T15:50:40.962 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:40.963 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj642 datafile 2019-12-21T15:50:41.004 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:41.005 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj643 datafile 2019-12-21T15:50:41.046 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:41.046 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj644 datafile 2019-12-21T15:50:41.155 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:41.155 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj645 datafile 2019-12-21T15:50:41.191 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:41.191 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj646 datafile 2019-12-21T15:50:41.288 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:41.289 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj647 datafile 2019-12-21T15:50:41.364 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:41.364 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj648 datafile 2019-12-21T15:50:41.430 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:41.431 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj649 datafile 2019-12-21T15:50:41.471 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:41.471 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj650 datafile 2019-12-21T15:50:41.521 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:41.521 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj651 datafile 2019-12-21T15:50:41.606 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:41.607 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj652 datafile 2019-12-21T15:50:41.646 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:41.646 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj653 datafile 2019-12-21T15:50:41.691 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:41.692 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj654 datafile 2019-12-21T15:50:41.790 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:41.790 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj655 datafile 2019-12-21T15:50:41.828 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:41.828 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj656 datafile 2019-12-21T15:50:41.907 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:41.907 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj657 datafile 2019-12-21T15:50:41.954 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:41.955 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj658 datafile 2019-12-21T15:50:41.996 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:41.996 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj659 datafile 2019-12-21T15:50:42.031 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:42.031 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj660 datafile 2019-12-21T15:50:42.071 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:42.071 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj661 datafile 2019-12-21T15:50:42.124 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:42.124 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj662 datafile 2019-12-21T15:50:42.163 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:42.163 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj663 datafile 2019-12-21T15:50:42.249 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:42.250 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj664 datafile 2019-12-21T15:50:42.288 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:42.288 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj665 datafile 2019-12-21T15:50:42.375 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:42.375 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj666 datafile 2019-12-21T15:50:42.501 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:42.501 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj667 datafile 2019-12-21T15:50:42.538 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:42.538 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj668 datafile 2019-12-21T15:50:42.668 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:42.668 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj669 datafile 2019-12-21T15:50:42.709 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:42.709 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj670 datafile 2019-12-21T15:50:42.755 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:42.755 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj671 datafile 2019-12-21T15:50:42.843 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:42.843 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj672 datafile 2019-12-21T15:50:42.888 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:42.888 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj673 datafile 2019-12-21T15:50:42.923 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:42.923 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj674 datafile 2019-12-21T15:50:42.958 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:42.959 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj675 datafile 2019-12-21T15:50:42.997 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:42.997 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj676 datafile 2019-12-21T15:50:43.038 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:43.038 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj677 datafile 2019-12-21T15:50:43.075 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:43.075 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj678 datafile 2019-12-21T15:50:43.120 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:43.120 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj679 datafile 2019-12-21T15:50:43.219 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:43.219 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj680 datafile 2019-12-21T15:50:43.263 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:43.263 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj681 datafile 2019-12-21T15:50:43.369 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:43.370 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj682 datafile 2019-12-21T15:50:43.413 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:43.414 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj683 datafile 2019-12-21T15:50:43.457 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:43.457 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj684 datafile 2019-12-21T15:50:43.519 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:43.520 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj685 datafile 2019-12-21T15:50:43.595 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:43.595 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj686 datafile 2019-12-21T15:50:43.662 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:43.662 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj687 datafile 2019-12-21T15:50:43.713 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:43.714 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj688 datafile 2019-12-21T15:50:43.756 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:43.756 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj689 datafile 2019-12-21T15:50:43.795 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:43.796 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj690 datafile 2019-12-21T15:50:43.879 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:43.879 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj691 datafile 2019-12-21T15:50:43.916 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:43.916 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj692 datafile 2019-12-21T15:50:43.996 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:43.996 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj693 datafile 2019-12-21T15:50:44.097 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:44.097 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj694 datafile 2019-12-21T15:50:44.142 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:44.143 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj695 datafile 2019-12-21T15:50:44.214 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:44.214 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj696 datafile 2019-12-21T15:50:44.259 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:44.259 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj697 datafile 2019-12-21T15:50:44.317 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:44.318 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj698 datafile 2019-12-21T15:50:44.381 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:44.381 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj699 datafile 2019-12-21T15:50:44.439 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:44.440 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj700 datafile 2019-12-21T15:50:44.477 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:44.478 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj701 datafile 2019-12-21T15:50:44.598 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:44.599 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj702 datafile 2019-12-21T15:50:44.666 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:44.666 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj703 datafile 2019-12-21T15:50:44.710 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:44.711 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj704 datafile 2019-12-21T15:50:44.768 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:44.768 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj705 datafile 2019-12-21T15:50:44.811 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:44.811 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj706 datafile 2019-12-21T15:50:44.899 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:44.899 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj707 datafile 2019-12-21T15:50:44.942 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:44.943 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj708 datafile 2019-12-21T15:50:45.025 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:45.025 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj709 datafile 2019-12-21T15:50:45.064 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:45.064 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj710 datafile 2019-12-21T15:50:45.151 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:45.151 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj711 datafile 2019-12-21T15:50:45.226 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:45.226 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj712 datafile 2019-12-21T15:50:45.267 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:45.268 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj713 datafile 2019-12-21T15:50:45.351 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:45.351 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj714 datafile 2019-12-21T15:50:45.396 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:45.397 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj715 datafile 2019-12-21T15:50:45.438 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:45.438 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj716 datafile 2019-12-21T15:50:45.481 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:45.481 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj717 datafile 2019-12-21T15:50:45.523 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:45.523 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj718 datafile 2019-12-21T15:50:45.568 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:45.568 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj719 datafile 2019-12-21T15:50:45.609 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:45.609 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj720 datafile 2019-12-21T15:50:45.693 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:45.693 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj721 datafile 2019-12-21T15:50:45.802 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:45.802 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj722 datafile 2019-12-21T15:50:45.886 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:45.886 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj723 datafile 2019-12-21T15:50:45.923 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:45.923 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj724 datafile 2019-12-21T15:50:46.003 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:46.003 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj725 datafile 2019-12-21T15:50:46.044 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:46.044 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj726 datafile 2019-12-21T15:50:46.086 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:46.086 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj727 datafile 2019-12-21T15:50:46.126 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:46.127 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj728 datafile 2019-12-21T15:50:46.175 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:46.175 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj729 datafile 2019-12-21T15:50:46.218 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:46.218 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj730 datafile 2019-12-21T15:50:46.295 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:46.295 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj731 datafile 2019-12-21T15:50:46.370 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:46.370 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj732 datafile 2019-12-21T15:50:46.406 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:46.406 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj733 datafile 2019-12-21T15:50:46.487 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:46.487 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj734 datafile 2019-12-21T15:50:46.535 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:46.535 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj735 datafile 2019-12-21T15:50:46.646 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:46.646 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj736 datafile 2019-12-21T15:50:46.693 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:46.693 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj737 datafile 2019-12-21T15:50:46.728 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:46.728 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj738 datafile 2019-12-21T15:50:46.772 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:46.772 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj739 datafile 2019-12-21T15:50:46.880 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:46.880 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj740 datafile 2019-12-21T15:50:46.926 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:46.927 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj741 datafile 2019-12-21T15:50:46.970 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:46.971 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj742 datafile 2019-12-21T15:50:47.008 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:47.008 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj743 datafile 2019-12-21T15:50:47.045 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:47.046 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj744 datafile 2019-12-21T15:50:47.130 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:47.131 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj745 datafile 2019-12-21T15:50:47.214 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:47.214 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj746 datafile 2019-12-21T15:50:47.323 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:47.323 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj747 datafile 2019-12-21T15:50:47.362 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:47.362 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj748 datafile 2019-12-21T15:50:47.405 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:47.405 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj749 datafile 2019-12-21T15:50:47.456 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:47.456 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj750 datafile 2019-12-21T15:50:47.523 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:47.524 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj751 datafile 2019-12-21T15:50:47.996 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:47.997 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj752 datafile 2019-12-21T15:50:48.367 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:48.367 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj753 datafile 2019-12-21T15:50:48.414 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:48.414 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj754 datafile 2019-12-21T15:50:48.475 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:48.475 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj755 datafile 2019-12-21T15:50:48.515 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:48.515 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj756 datafile 2019-12-21T15:50:48.634 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:48.634 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj757 datafile 2019-12-21T15:50:48.719 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:48.719 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj758 datafile 2019-12-21T15:50:48.757 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:48.757 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj759 datafile 2019-12-21T15:50:48.795 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:48.795 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj760 datafile 2019-12-21T15:50:48.851 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:48.851 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj761 datafile 2019-12-21T15:50:48.888 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:48.888 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj762 datafile 2019-12-21T15:50:48.925 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:48.925 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj763 datafile 2019-12-21T15:50:49.001 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:49.002 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj764 datafile 2019-12-21T15:50:49.085 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:49.085 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj765 datafile 2019-12-21T15:50:49.169 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:49.169 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj766 datafile 2019-12-21T15:50:49.207 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:49.207 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj767 datafile 2019-12-21T15:50:49.272 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:49.272 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj768 datafile 2019-12-21T15:50:49.386 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:49.387 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj769 datafile 2019-12-21T15:50:49.424 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:49.424 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj770 datafile 2019-12-21T15:50:49.503 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:49.504 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj771 datafile 2019-12-21T15:50:49.542 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:49.542 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj772 datafile 2019-12-21T15:50:49.586 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:49.586 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj773 datafile 2019-12-21T15:50:49.624 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:49.624 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj774 datafile 2019-12-21T15:50:49.704 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:49.704 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj775 datafile 2019-12-21T15:50:49.758 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:49.759 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj776 datafile 2019-12-21T15:50:49.846 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:49.846 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj777 datafile 2019-12-21T15:50:49.884 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:49.884 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj778 datafile 2019-12-21T15:50:49.988 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:49.988 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj779 datafile 2019-12-21T15:50:50.025 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:50.026 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj780 datafile 2019-12-21T15:50:50.105 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:50.105 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj781 datafile 2019-12-21T15:50:50.142 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:50.142 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj782 datafile 2019-12-21T15:50:50.177 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:50.178 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj783 datafile 2019-12-21T15:50:50.220 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:50.221 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj784 datafile 2019-12-21T15:50:50.289 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:50.289 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj785 datafile 2019-12-21T15:50:50.328 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:50.328 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj786 datafile 2019-12-21T15:50:50.367 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:50.367 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj787 datafile 2019-12-21T15:50:50.409 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:50.409 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj788 datafile 2019-12-21T15:50:50.489 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:50.489 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj789 datafile 2019-12-21T15:50:50.565 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:50.565 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj790 datafile 2019-12-21T15:50:50.642 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:50.642 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj791 datafile 2019-12-21T15:50:50.765 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:50.766 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj792 datafile 2019-12-21T15:50:50.849 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:50.850 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj793 datafile 2019-12-21T15:50:50.941 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:50.942 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj794 datafile 2019-12-21T15:50:50.984 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:50.984 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj795 datafile 2019-12-21T15:50:51.019 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:51.020 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj796 datafile 2019-12-21T15:50:51.092 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:51.092 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj797 datafile 2019-12-21T15:50:51.134 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:51.134 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj798 datafile 2019-12-21T15:50:51.195 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:51.195 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj799 datafile 2019-12-21T15:50:51.250 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:51.250 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj800 datafile 2019-12-21T15:50:51.294 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:51.295 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj801 datafile 2019-12-21T15:50:51.334 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:51.334 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj802 datafile 2019-12-21T15:50:51.379 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:51.379 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj803 datafile 2019-12-21T15:50:51.484 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:51.485 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj804 datafile 2019-12-21T15:50:51.526 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:51.526 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj805 datafile 2019-12-21T15:50:51.609 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:51.610 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj806 datafile 2019-12-21T15:50:51.655 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:51.656 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj807 datafile 2019-12-21T15:50:51.735 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:51.735 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj808 datafile 2019-12-21T15:50:51.779 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:51.779 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj809 datafile 2019-12-21T15:50:51.829 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:51.829 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj810 datafile 2019-12-21T15:50:51.885 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:51.885 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj811 datafile 2019-12-21T15:50:51.969 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:51.969 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj812 datafile 2019-12-21T15:50:52.078 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:52.078 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj813 datafile 2019-12-21T15:50:52.121 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:52.121 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj814 datafile 2019-12-21T15:50:52.195 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:52.195 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj815 datafile 2019-12-21T15:50:52.239 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:52.239 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj816 datafile 2019-12-21T15:50:52.275 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:52.275 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj817 datafile 2019-12-21T15:50:52.317 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:52.318 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj818 datafile 2019-12-21T15:50:52.359 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:52.360 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj819 datafile 2019-12-21T15:50:52.404 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:52.404 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj820 datafile 2019-12-21T15:50:52.462 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:52.462 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj821 datafile 2019-12-21T15:50:52.500 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:52.500 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj822 datafile 2019-12-21T15:50:52.637 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:52.638 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj823 datafile 2019-12-21T15:50:52.738 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:52.739 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj824 datafile 2019-12-21T15:50:52.813 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:52.814 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj825 datafile 2019-12-21T15:50:52.850 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:52.850 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj826 datafile 2019-12-21T15:50:52.893 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:52.893 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj827 datafile 2019-12-21T15:50:52.934 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:52.935 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj828 datafile 2019-12-21T15:50:52.976 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:52.976 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj829 datafile 2019-12-21T15:50:53.018 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:53.018 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj830 datafile 2019-12-21T15:50:53.105 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:53.106 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj831 datafile 2019-12-21T15:50:53.173 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:53.173 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj832 datafile 2019-12-21T15:50:53.213 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:53.213 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj833 datafile 2019-12-21T15:50:53.323 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:53.323 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj834 datafile 2019-12-21T15:50:53.361 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:53.362 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj835 datafile 2019-12-21T15:50:53.403 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:53.403 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj836 datafile 2019-12-21T15:50:53.482 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:53.482 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj837 datafile 2019-12-21T15:50:53.520 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:53.520 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj838 datafile 2019-12-21T15:50:53.666 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:53.666 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj839 datafile 2019-12-21T15:50:53.741 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:53.742 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj840 datafile 2019-12-21T15:50:53.778 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:53.779 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj841 datafile 2019-12-21T15:50:53.818 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:53.818 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj842 datafile 2019-12-21T15:50:53.900 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:53.900 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj843 datafile 2019-12-21T15:50:53.992 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:53.992 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj844 datafile 2019-12-21T15:50:54.075 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:54.075 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj845 datafile 2019-12-21T15:50:54.120 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:54.120 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj846 datafile 2019-12-21T15:50:54.192 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:54.192 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj847 datafile 2019-12-21T15:50:54.231 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:54.231 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj848 datafile 2019-12-21T15:50:54.267 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:54.268 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj849 datafile 2019-12-21T15:50:54.376 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:54.376 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj850 datafile 2019-12-21T15:50:54.414 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:54.414 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj851 datafile 2019-12-21T15:50:54.485 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:54.485 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj852 datafile 2019-12-21T15:50:54.527 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:54.527 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj853 datafile 2019-12-21T15:50:54.572 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:54.572 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj854 datafile 2019-12-21T15:50:54.711 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:54.711 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj855 datafile 2019-12-21T15:50:54.794 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:54.794 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj856 datafile 2019-12-21T15:50:54.835 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:54.835 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj857 datafile 2019-12-21T15:50:54.879 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:54.880 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj858 datafile 2019-12-21T15:50:54.919 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:54.920 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj859 datafile 2019-12-21T15:50:55.020 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:55.020 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj860 datafile 2019-12-21T15:50:55.060 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:55.060 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj861 datafile 2019-12-21T15:50:55.104 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:55.104 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj862 datafile 2019-12-21T15:50:55.143 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:55.143 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj863 datafile 2019-12-21T15:50:55.185 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:55.185 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj864 datafile 2019-12-21T15:50:55.227 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:55.227 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj865 datafile 2019-12-21T15:50:55.303 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:55.303 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj866 datafile 2019-12-21T15:50:55.395 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:55.396 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj867 datafile 2019-12-21T15:50:55.454 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:55.454 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj868 datafile 2019-12-21T15:50:55.496 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:55.496 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj869 datafile 2019-12-21T15:50:55.531 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:55.531 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj870 datafile 2019-12-21T15:50:55.568 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:55.568 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj871 datafile 2019-12-21T15:50:55.613 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:55.613 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj872 datafile 2019-12-21T15:50:55.654 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:55.654 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj873 datafile 2019-12-21T15:50:55.738 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:55.738 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj874 datafile 2019-12-21T15:50:55.830 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:55.830 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj875 datafile 2019-12-21T15:50:55.914 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:55.914 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj876 datafile 2019-12-21T15:50:56.006 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:56.006 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj877 datafile 2019-12-21T15:50:56.043 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:56.043 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj878 datafile 2019-12-21T15:50:56.096 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:56.096 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj879 datafile 2019-12-21T15:50:56.135 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:56.135 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj880 datafile 2019-12-21T15:50:56.177 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:56.177 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj881 datafile 2019-12-21T15:50:56.221 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:56.221 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj882 datafile 2019-12-21T15:50:56.314 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:56.315 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj883 datafile 2019-12-21T15:50:56.408 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:56.408 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj884 datafile 2019-12-21T15:50:56.482 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:56.482 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj885 datafile 2019-12-21T15:50:56.527 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:56.527 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj886 datafile 2019-12-21T15:50:56.569 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:56.569 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj887 datafile 2019-12-21T15:50:56.640 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:56.640 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj888 datafile 2019-12-21T15:50:56.725 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:56.725 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj889 datafile 2019-12-21T15:50:56.765 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:56.765 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj890 datafile 2019-12-21T15:50:56.858 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:56.858 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj891 datafile 2019-12-21T15:50:56.933 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:56.933 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj892 datafile 2019-12-21T15:50:56.977 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:56.977 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj893 datafile 2019-12-21T15:50:57.051 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:57.051 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj894 datafile 2019-12-21T15:50:57.187 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:57.188 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj895 datafile 2019-12-21T15:50:57.243 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:57.243 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj896 datafile 2019-12-21T15:50:57.287 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:57.287 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj897 datafile 2019-12-21T15:50:57.327 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:57.328 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj898 datafile 2019-12-21T15:50:57.362 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:57.362 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj899 datafile 2019-12-21T15:50:57.403 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:57.403 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj900 datafile 2019-12-21T15:50:57.444 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:57.445 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj901 datafile 2019-12-21T15:50:57.498 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:57.499 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj902 datafile 2019-12-21T15:50:57.602 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:57.602 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj903 datafile 2019-12-21T15:50:57.686 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:57.686 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj904 datafile 2019-12-21T15:50:57.728 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:57.728 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj905 datafile 2019-12-21T15:50:57.773 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:57.773 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj906 datafile 2019-12-21T15:50:57.811 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:57.811 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj907 datafile 2019-12-21T15:50:57.853 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:57.853 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj908 datafile 2019-12-21T15:50:57.895 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:57.896 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj909 datafile 2019-12-21T15:50:57.944 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:57.944 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj910 datafile 2019-12-21T15:50:57.991 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:57.991 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj911 datafile 2019-12-21T15:50:58.029 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:58.029 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj912 datafile 2019-12-21T15:50:58.120 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:58.120 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj913 datafile 2019-12-21T15:50:58.171 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:58.171 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj914 datafile 2019-12-21T15:50:58.220 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:58.220 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj915 datafile 2019-12-21T15:50:58.257 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:58.257 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj916 datafile 2019-12-21T15:50:58.290 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:58.290 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj917 datafile 2019-12-21T15:50:58.353 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:58.353 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj918 datafile 2019-12-21T15:50:58.395 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:58.396 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj919 datafile 2019-12-21T15:50:58.504 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:58.505 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj920 datafile 2019-12-21T15:50:58.546 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:58.546 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj921 datafile 2019-12-21T15:50:58.646 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:58.647 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj922 datafile 2019-12-21T15:50:58.696 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:58.696 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj923 datafile 2019-12-21T15:50:58.733 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:58.733 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj924 datafile 2019-12-21T15:50:58.779 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:58.779 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj925 datafile 2019-12-21T15:50:58.989 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:58.989 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj926 datafile 2019-12-21T15:50:59.030 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:59.030 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj927 datafile 2019-12-21T15:50:59.073 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:59.073 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj928 datafile 2019-12-21T15:50:59.115 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:59.115 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj929 datafile 2019-12-21T15:50:59.189 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:59.189 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj930 datafile 2019-12-21T15:50:59.232 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:59.233 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj931 datafile 2019-12-21T15:50:59.298 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:59.298 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj932 datafile 2019-12-21T15:50:59.353 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:59.353 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj933 datafile 2019-12-21T15:50:59.499 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:59.499 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj934 datafile 2019-12-21T15:50:59.538 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:59.539 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj935 datafile 2019-12-21T15:50:59.574 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:59.574 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj936 datafile 2019-12-21T15:50:59.615 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:59.615 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj937 datafile 2019-12-21T15:50:59.707 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:59.708 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj938 datafile 2019-12-21T15:50:59.745 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:59.745 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj939 datafile 2019-12-21T15:50:59.796 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:59.796 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj940 datafile 2019-12-21T15:50:59.916 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:50:59.916 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj941 datafile 2019-12-21T15:51:00.008 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:00.009 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj942 datafile 2019-12-21T15:51:00.101 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:00.101 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj943 datafile 2019-12-21T15:51:00.193 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:00.193 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj944 datafile 2019-12-21T15:51:00.237 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:00.238 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj945 datafile 2019-12-21T15:51:00.273 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:00.274 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj946 datafile 2019-12-21T15:51:00.321 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:00.321 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj947 datafile 2019-12-21T15:51:00.366 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:00.366 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj948 datafile 2019-12-21T15:51:00.413 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:00.413 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj949 datafile 2019-12-21T15:51:00.450 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:00.451 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj950 datafile 2019-12-21T15:51:00.526 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:00.526 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj951 datafile 2019-12-21T15:51:00.565 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:00.566 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj952 datafile 2019-12-21T15:51:00.638 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:00.638 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj953 datafile 2019-12-21T15:51:00.678 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:00.679 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj954 datafile 2019-12-21T15:51:00.760 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:00.761 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj955 datafile 2019-12-21T15:51:00.878 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:00.878 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj956 datafile 2019-12-21T15:51:00.953 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:00.953 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj957 datafile 2019-12-21T15:51:01.016 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:01.016 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj958 datafile 2019-12-21T15:51:01.087 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:01.087 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj959 datafile 2019-12-21T15:51:01.132 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:01.133 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj960 datafile 2019-12-21T15:51:01.171 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:01.171 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj961 datafile 2019-12-21T15:51:01.212 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:01.212 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj962 datafile 2019-12-21T15:51:01.254 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:01.254 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj963 datafile 2019-12-21T15:51:01.337 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:01.338 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj964 datafile 2019-12-21T15:51:01.382 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:01.383 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj965 datafile 2019-12-21T15:51:01.437 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:01.437 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj966 datafile 2019-12-21T15:51:01.496 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:01.496 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj967 datafile 2019-12-21T15:51:01.571 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:01.571 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj968 datafile 2019-12-21T15:51:01.616 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:01.616 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj969 datafile 2019-12-21T15:51:01.654 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:01.654 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj970 datafile 2019-12-21T15:51:01.763 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:01.763 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj971 datafile 2019-12-21T15:51:01.864 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:01.865 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj972 datafile 2019-12-21T15:51:01.948 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:01.948 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj973 datafile 2019-12-21T15:51:01.991 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:01.991 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj974 datafile 2019-12-21T15:51:02.032 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:02.032 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj975 datafile 2019-12-21T15:51:02.114 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:02.115 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj976 datafile 2019-12-21T15:51:02.149 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:02.149 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj977 datafile 2019-12-21T15:51:02.187 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:02.188 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj978 datafile 2019-12-21T15:51:02.230 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:02.230 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj979 datafile 2019-12-21T15:51:02.271 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:02.271 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj980 datafile 2019-12-21T15:51:02.373 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:02.374 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj981 datafile 2019-12-21T15:51:02.432 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:02.432 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj982 datafile 2019-12-21T15:51:02.516 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:02.516 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj983 datafile 2019-12-21T15:51:02.557 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:02.558 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj984 datafile 2019-12-21T15:51:02.700 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:02.700 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj985 datafile 2019-12-21T15:51:02.754 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:02.755 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj986 datafile 2019-12-21T15:51:02.791 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:02.791 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj987 datafile 2019-12-21T15:51:02.833 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:02.833 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj988 datafile 2019-12-21T15:51:02.871 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:02.871 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj989 datafile 2019-12-21T15:51:02.912 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:02.913 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj990 datafile 2019-12-21T15:51:02.975 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:02.975 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj991 datafile 2019-12-21T15:51:03.067 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:03.067 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj992 datafile 2019-12-21T15:51:03.167 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:03.168 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj993 datafile 2019-12-21T15:51:03.221 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:03.221 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj994 datafile 2019-12-21T15:51:03.301 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:03.301 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj995 datafile 2019-12-21T15:51:03.346 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:03.346 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj996 datafile 2019-12-21T15:51:03.396 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:03.396 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj997 datafile 2019-12-21T15:51:03.477 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:03.477 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj998 datafile 2019-12-21T15:51:03.552 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:03.552 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj999 datafile 2019-12-21T15:51:03.604 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:03.605 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test2 put obj1000 datafile 2019-12-21T15:51:03.694 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:72: TEST_recover_unexpected: for i in $(seq 1 $POOLS) 2019-12-21T15:51:03.695 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: seq 1 1000 2019-12-21T15:51:03.696 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:03.696 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj1 datafile 2019-12-21T15:51:03.746 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:03.746 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj2 datafile 2019-12-21T15:51:03.853 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:03.853 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj3 datafile 2019-12-21T15:51:03.902 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:03.902 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj4 datafile 2019-12-21T15:51:03.938 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:03.938 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj5 datafile 2019-12-21T15:51:04.020 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:04.020 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj6 datafile 2019-12-21T15:51:04.103 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:04.104 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj7 datafile 2019-12-21T15:51:04.148 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:04.148 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj8 datafile 2019-12-21T15:51:04.189 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:04.190 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj9 datafile 2019-12-21T15:51:04.238 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:04.238 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj10 datafile 2019-12-21T15:51:04.280 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:04.280 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj11 datafile 2019-12-21T15:51:04.362 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:04.363 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj12 datafile 2019-12-21T15:51:04.448 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:04.448 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj13 datafile 2019-12-21T15:51:04.554 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:04.555 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj14 datafile 2019-12-21T15:51:04.622 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:04.622 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj15 datafile 2019-12-21T15:51:04.665 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:04.665 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj16 datafile 2019-12-21T15:51:04.747 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:04.747 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj17 datafile 2019-12-21T15:51:04.788 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:04.788 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj18 datafile 2019-12-21T15:51:04.830 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:04.830 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj19 datafile 2019-12-21T15:51:04.914 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:04.914 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj20 datafile 2019-12-21T15:51:04.997 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:04.998 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj21 datafile 2019-12-21T15:51:05.038 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:05.038 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj22 datafile 2019-12-21T15:51:05.156 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:05.157 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj23 datafile 2019-12-21T15:51:05.197 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:05.197 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj24 datafile 2019-12-21T15:51:05.246 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:05.246 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj25 datafile 2019-12-21T15:51:05.315 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:05.315 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj26 datafile 2019-12-21T15:51:05.363 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:05.364 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj27 datafile 2019-12-21T15:51:05.461 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:05.461 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj28 datafile 2019-12-21T15:51:05.506 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:05.506 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj29 datafile 2019-12-21T15:51:05.582 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:05.583 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj30 datafile 2019-12-21T15:51:05.624 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:05.625 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj31 datafile 2019-12-21T15:51:05.665 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:05.665 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj32 datafile 2019-12-21T15:51:05.750 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:05.750 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj33 datafile 2019-12-21T15:51:05.833 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:05.833 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj34 datafile 2019-12-21T15:51:05.916 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:05.916 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj35 datafile 2019-12-21T15:51:05.984 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:05.984 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj36 datafile 2019-12-21T15:51:06.030 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:06.030 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj37 datafile 2019-12-21T15:51:06.109 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:06.109 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj38 datafile 2019-12-21T15:51:06.149 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:06.149 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj39 datafile 2019-12-21T15:51:06.226 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:06.226 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj40 datafile 2019-12-21T15:51:06.271 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:06.272 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj41 datafile 2019-12-21T15:51:06.315 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:06.315 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj42 datafile 2019-12-21T15:51:06.401 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:06.402 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj43 datafile 2019-12-21T15:51:06.447 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:06.447 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj44 datafile 2019-12-21T15:51:06.491 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:06.491 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj45 datafile 2019-12-21T15:51:06.586 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:06.586 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj46 datafile 2019-12-21T15:51:06.663 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:06.663 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj47 datafile 2019-12-21T15:51:06.778 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:06.779 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj48 datafile 2019-12-21T15:51:06.862 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:06.862 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj49 datafile 2019-12-21T15:51:06.914 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:06.914 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj50 datafile 2019-12-21T15:51:06.964 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:06.964 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj51 datafile 2019-12-21T15:51:07.045 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:07.046 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj52 datafile 2019-12-21T15:51:07.088 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:07.089 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj53 datafile 2019-12-21T15:51:07.179 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:07.179 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj54 datafile 2019-12-21T15:51:07.255 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:07.255 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj55 datafile 2019-12-21T15:51:07.298 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:07.298 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj56 datafile 2019-12-21T15:51:07.397 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:07.397 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj57 datafile 2019-12-21T15:51:07.438 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:07.438 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj58 datafile 2019-12-21T15:51:07.489 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:07.489 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj59 datafile 2019-12-21T15:51:07.547 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:07.547 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj60 datafile 2019-12-21T15:51:07.589 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:07.589 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj61 datafile 2019-12-21T15:51:07.631 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:07.631 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj62 datafile 2019-12-21T15:51:07.673 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:07.673 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj63 datafile 2019-12-21T15:51:07.773 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:07.773 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj64 datafile 2019-12-21T15:51:07.808 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:07.808 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj65 datafile 2019-12-21T15:51:07.890 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:07.890 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj66 datafile 2019-12-21T15:51:07.931 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:07.931 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj67 datafile 2019-12-21T15:51:07.972 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:07.973 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj68 datafile 2019-12-21T15:51:08.015 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:08.015 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj69 datafile 2019-12-21T15:51:08.073 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:08.073 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj70 datafile 2019-12-21T15:51:08.108 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:08.108 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj71 datafile 2019-12-21T15:51:08.198 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:08.199 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj72 datafile 2019-12-21T15:51:08.291 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:08.291 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj73 datafile 2019-12-21T15:51:08.374 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:08.374 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj74 datafile 2019-12-21T15:51:08.415 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:08.415 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj75 datafile 2019-12-21T15:51:08.517 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:08.517 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj76 datafile 2019-12-21T15:51:08.596 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:08.596 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj77 datafile 2019-12-21T15:51:08.640 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:08.640 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj78 datafile 2019-12-21T15:51:08.681 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:08.681 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj79 datafile 2019-12-21T15:51:08.723 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:08.723 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj80 datafile 2019-12-21T15:51:08.773 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:08.773 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj81 datafile 2019-12-21T15:51:08.815 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:08.815 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj82 datafile 2019-12-21T15:51:08.909 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:08.910 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj83 datafile 2019-12-21T15:51:08.956 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:08.956 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj84 datafile 2019-12-21T15:51:09.018 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:09.018 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj85 datafile 2019-12-21T15:51:09.098 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:09.099 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj86 datafile 2019-12-21T15:51:09.189 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:09.190 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj87 datafile 2019-12-21T15:51:09.231 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:09.231 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj88 datafile 2019-12-21T15:51:09.290 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:09.290 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj89 datafile 2019-12-21T15:51:09.369 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:09.369 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj90 datafile 2019-12-21T15:51:09.415 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:09.415 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj91 datafile 2019-12-21T15:51:09.570 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:09.570 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj92 datafile 2019-12-21T15:51:09.654 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:09.654 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj93 datafile 2019-12-21T15:51:09.708 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:09.708 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj94 datafile 2019-12-21T15:51:09.747 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:09.747 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj95 datafile 2019-12-21T15:51:09.790 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:09.790 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj96 datafile 2019-12-21T15:51:09.833 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:09.834 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj97 datafile 2019-12-21T15:51:09.884 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:09.884 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj98 datafile 2019-12-21T15:51:09.937 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:09.937 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj99 datafile 2019-12-21T15:51:10.038 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:10.038 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj100 datafile 2019-12-21T15:51:10.122 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:10.122 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj101 datafile 2019-12-21T15:51:10.173 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:10.174 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj102 datafile 2019-12-21T15:51:10.264 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:10.264 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj103 datafile 2019-12-21T15:51:10.308 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:10.308 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj104 datafile 2019-12-21T15:51:10.347 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:10.349 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj105 datafile 2019-12-21T15:51:10.390 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:10.390 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj106 datafile 2019-12-21T15:51:10.433 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:10.434 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj107 datafile 2019-12-21T15:51:10.522 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:10.559 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj108 datafile 2019-12-21T15:51:10.648 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:10.648 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj109 datafile 2019-12-21T15:51:10.692 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:10.692 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj110 datafile 2019-12-21T15:51:10.732 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:10.732 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj111 datafile 2019-12-21T15:51:10.783 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:10.784 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj112 datafile 2019-12-21T15:51:10.899 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:10.899 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj113 datafile 2019-12-21T15:51:10.942 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:10.942 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj114 datafile 2019-12-21T15:51:11.032 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:11.033 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj115 datafile 2019-12-21T15:51:11.116 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:11.116 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj116 datafile 2019-12-21T15:51:11.157 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:11.157 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj117 datafile 2019-12-21T15:51:11.233 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:11.233 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj118 datafile 2019-12-21T15:51:11.274 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:11.274 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj119 datafile 2019-12-21T15:51:11.341 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:11.342 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj120 datafile 2019-12-21T15:51:11.382 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:11.383 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj121 datafile 2019-12-21T15:51:11.424 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:11.424 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj122 datafile 2019-12-21T15:51:11.509 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:11.509 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj123 datafile 2019-12-21T15:51:11.617 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:11.617 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj124 datafile 2019-12-21T15:51:11.667 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:11.667 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj125 datafile 2019-12-21T15:51:11.768 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:11.768 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj126 datafile 2019-12-21T15:51:11.852 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:11.852 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj127 datafile 2019-12-21T15:51:11.892 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:11.892 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj128 datafile 2019-12-21T15:51:11.941 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:11.941 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj129 datafile 2019-12-21T15:51:11.982 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:11.983 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj130 datafile 2019-12-21T15:51:12.024 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:12.025 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj131 datafile 2019-12-21T15:51:12.102 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:12.102 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj132 datafile 2019-12-21T15:51:12.149 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:12.149 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj133 datafile 2019-12-21T15:51:12.192 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:12.192 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj134 datafile 2019-12-21T15:51:12.302 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:12.303 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj135 datafile 2019-12-21T15:51:12.349 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:12.350 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj136 datafile 2019-12-21T15:51:12.391 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:12.391 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj137 datafile 2019-12-21T15:51:12.433 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:12.433 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj138 datafile 2019-12-21T15:51:12.474 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:12.474 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj139 datafile 2019-12-21T15:51:12.578 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:12.578 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj140 datafile 2019-12-21T15:51:12.687 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:12.687 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj141 datafile 2019-12-21T15:51:12.734 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:12.734 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj142 datafile 2019-12-21T15:51:12.775 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:12.776 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj143 datafile 2019-12-21T15:51:12.817 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:12.817 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj144 datafile 2019-12-21T15:51:12.859 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:12.859 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj145 datafile 2019-12-21T15:51:12.946 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:12.946 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj146 datafile 2019-12-21T15:51:12.992 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:12.992 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj147 datafile 2019-12-21T15:51:13.088 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:13.088 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj148 datafile 2019-12-21T15:51:13.180 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:13.180 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj149 datafile 2019-12-21T15:51:13.233 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:13.233 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj150 datafile 2019-12-21T15:51:13.283 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:13.284 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj151 datafile 2019-12-21T15:51:13.334 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:13.334 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj152 datafile 2019-12-21T15:51:13.405 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:13.405 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj153 datafile 2019-12-21T15:51:13.461 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:13.461 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj154 datafile 2019-12-21T15:51:13.500 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:13.500 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj155 datafile 2019-12-21T15:51:13.542 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:13.542 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj156 datafile 2019-12-21T15:51:13.583 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:13.584 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj157 datafile 2019-12-21T15:51:13.695 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:13.696 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj158 datafile 2019-12-21T15:51:13.742 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:13.742 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj159 datafile 2019-12-21T15:51:13.783 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:13.783 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj160 datafile 2019-12-21T15:51:13.865 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:13.865 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj161 datafile 2019-12-21T15:51:13.957 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:13.957 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj162 datafile 2019-12-21T15:51:14.041 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:14.041 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj163 datafile 2019-12-21T15:51:14.084 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:14.084 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj164 datafile 2019-12-21T15:51:14.240 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:14.240 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj165 datafile 2019-12-21T15:51:14.300 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:14.301 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj166 datafile 2019-12-21T15:51:14.341 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:14.341 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj167 datafile 2019-12-21T15:51:14.416 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:14.416 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj168 datafile 2019-12-21T15:51:14.525 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:14.525 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj169 datafile 2019-12-21T15:51:14.568 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:14.568 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj170 datafile 2019-12-21T15:51:14.651 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:14.651 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj171 datafile 2019-12-21T15:51:14.734 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:14.734 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj172 datafile 2019-12-21T15:51:14.819 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:14.820 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj173 datafile 2019-12-21T15:51:14.859 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:14.859 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj174 datafile 2019-12-21T15:51:14.900 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:14.900 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj175 datafile 2019-12-21T15:51:14.950 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:14.951 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj176 datafile 2019-12-21T15:51:15.035 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:15.035 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj177 datafile 2019-12-21T15:51:15.099 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:15.099 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj178 datafile 2019-12-21T15:51:15.177 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:15.177 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj179 datafile 2019-12-21T15:51:15.244 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:15.244 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj180 datafile 2019-12-21T15:51:15.311 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:15.311 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj181 datafile 2019-12-21T15:51:15.351 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:15.351 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj182 datafile 2019-12-21T15:51:15.445 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:15.445 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj183 datafile 2019-12-21T15:51:15.491 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:15.491 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj184 datafile 2019-12-21T15:51:15.578 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:15.579 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj185 datafile 2019-12-21T15:51:15.627 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:15.627 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj186 datafile 2019-12-21T15:51:15.704 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:15.704 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj187 datafile 2019-12-21T15:51:15.759 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:15.759 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj188 datafile 2019-12-21T15:51:15.818 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:15.818 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj189 datafile 2019-12-21T15:51:15.904 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:15.905 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj190 datafile 2019-12-21T15:51:15.996 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:15.996 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj191 datafile 2019-12-21T15:51:16.080 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:16.080 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj192 datafile 2019-12-21T15:51:16.124 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:16.124 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj193 datafile 2019-12-21T15:51:16.166 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:16.166 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj194 datafile 2019-12-21T15:51:16.256 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:16.256 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj195 datafile 2019-12-21T15:51:16.300 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:16.301 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj196 datafile 2019-12-21T15:51:16.342 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:16.342 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj197 datafile 2019-12-21T15:51:16.384 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:16.384 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj198 datafile 2019-12-21T15:51:16.424 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:16.425 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj199 datafile 2019-12-21T15:51:16.539 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:16.540 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj200 datafile 2019-12-21T15:51:16.585 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:16.585 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj201 datafile 2019-12-21T15:51:16.682 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:16.682 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj202 datafile 2019-12-21T15:51:16.765 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:16.766 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj203 datafile 2019-12-21T15:51:16.840 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:16.841 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj204 datafile 2019-12-21T15:51:16.884 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:16.884 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj205 datafile 2019-12-21T15:51:16.924 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:16.925 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj206 datafile 2019-12-21T15:51:16.967 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:16.967 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj207 datafile 2019-12-21T15:51:17.008 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:17.008 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj208 datafile 2019-12-21T15:51:17.091 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:17.092 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj209 datafile 2019-12-21T15:51:17.200 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:17.200 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj210 datafile 2019-12-21T15:51:17.241 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:17.241 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj211 datafile 2019-12-21T15:51:17.342 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:17.342 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj212 datafile 2019-12-21T15:51:17.383 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:17.383 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj213 datafile 2019-12-21T15:51:17.424 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:17.425 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj214 datafile 2019-12-21T15:51:17.466 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:17.466 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj215 datafile 2019-12-21T15:51:17.551 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:17.551 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj216 datafile 2019-12-21T15:51:17.592 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:17.592 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj217 datafile 2019-12-21T15:51:17.634 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:17.634 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj218 datafile 2019-12-21T15:51:17.718 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:17.718 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj219 datafile 2019-12-21T15:51:17.767 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:17.768 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj220 datafile 2019-12-21T15:51:17.869 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:17.869 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj221 datafile 2019-12-21T15:51:17.977 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:17.977 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj222 datafile 2019-12-21T15:51:18.026 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:18.026 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj223 datafile 2019-12-21T15:51:18.068 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:18.068 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj224 datafile 2019-12-21T15:51:18.108 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:18.109 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj225 datafile 2019-12-21T15:51:18.158 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:18.158 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj226 datafile 2019-12-21T15:51:18.244 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:18.244 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj227 datafile 2019-12-21T15:51:18.282 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:18.282 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj228 datafile 2019-12-21T15:51:18.326 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:18.326 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj229 datafile 2019-12-21T15:51:18.368 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:18.368 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj230 datafile 2019-12-21T15:51:18.455 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:18.455 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj231 datafile 2019-12-21T15:51:18.490 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:18.490 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj232 datafile 2019-12-21T15:51:18.534 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:18.535 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj233 datafile 2019-12-21T15:51:18.575 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:18.575 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj234 datafile 2019-12-21T15:51:18.729 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:18.729 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj235 datafile 2019-12-21T15:51:18.783 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:18.783 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj236 datafile 2019-12-21T15:51:18.825 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:18.825 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj237 datafile 2019-12-21T15:51:18.913 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:18.913 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj238 datafile 2019-12-21T15:51:18.958 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:18.959 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj239 datafile 2019-12-21T15:51:19.038 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:19.038 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj240 datafile 2019-12-21T15:51:19.083 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:19.084 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj241 datafile 2019-12-21T15:51:19.180 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:19.180 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj242 datafile 2019-12-21T15:51:19.226 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:19.226 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj243 datafile 2019-12-21T15:51:19.291 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:19.291 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj244 datafile 2019-12-21T15:51:19.461 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:19.462 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj245 datafile 2019-12-21T15:51:20.074 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:20.074 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj246 datafile 2019-12-21T15:51:20.117 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:20.118 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj247 datafile 2019-12-21T15:51:20.157 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:20.158 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj248 datafile 2019-12-21T15:51:20.199 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:20.199 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj249 datafile 2019-12-21T15:51:20.290 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:20.290 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj250 datafile 2019-12-21T15:51:20.374 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:20.374 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj251 datafile 2019-12-21T15:51:20.474 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:20.474 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj252 datafile 2019-12-21T15:51:20.558 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:20.558 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj253 datafile 2019-12-21T15:51:20.643 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:20.643 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj254 datafile 2019-12-21T15:51:20.684 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:20.685 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj255 datafile 2019-12-21T15:51:20.734 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:20.734 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj256 datafile 2019-12-21T15:51:20.776 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:20.776 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj257 datafile 2019-12-21T15:51:20.826 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:20.826 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj258 datafile 2019-12-21T15:51:20.908 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:20.909 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj259 datafile 2019-12-21T15:51:20.961 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:20.961 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj260 datafile 2019-12-21T15:51:21.017 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:21.017 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj261 datafile 2019-12-21T15:51:21.126 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:21.126 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj262 datafile 2019-12-21T15:51:21.168 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:21.168 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj263 datafile 2019-12-21T15:51:21.260 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:21.260 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj264 datafile 2019-12-21T15:51:21.300 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:21.300 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj265 datafile 2019-12-21T15:51:21.341 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:21.341 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj266 datafile 2019-12-21T15:51:21.383 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:21.383 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj267 datafile 2019-12-21T15:51:21.493 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:21.493 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj268 datafile 2019-12-21T15:51:21.543 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:21.543 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj269 datafile 2019-12-21T15:51:21.627 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:21.627 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj270 datafile 2019-12-21T15:51:21.667 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:21.668 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj271 datafile 2019-12-21T15:51:21.753 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:21.753 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj272 datafile 2019-12-21T15:51:21.792 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:21.793 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj273 datafile 2019-12-21T15:51:21.833 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:21.833 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj274 datafile 2019-12-21T15:51:21.885 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:21.885 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj275 datafile 2019-12-21T15:51:21.925 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:21.925 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj276 datafile 2019-12-21T15:51:22.003 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:22.003 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj277 datafile 2019-12-21T15:51:22.087 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:22.087 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj278 datafile 2019-12-21T15:51:22.187 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:22.187 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj279 datafile 2019-12-21T15:51:22.262 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:22.262 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj280 datafile 2019-12-21T15:51:22.308 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:22.309 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj281 datafile 2019-12-21T15:51:22.358 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:22.358 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj282 datafile 2019-12-21T15:51:22.454 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:22.455 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj283 datafile 2019-12-21T15:51:22.491 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:22.492 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj284 datafile 2019-12-21T15:51:22.534 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:22.534 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj285 datafile 2019-12-21T15:51:22.605 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:22.605 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj286 datafile 2019-12-21T15:51:22.667 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:22.667 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj287 datafile 2019-12-21T15:51:22.747 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:22.748 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj288 datafile 2019-12-21T15:51:22.864 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:22.865 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj289 datafile 2019-12-21T15:51:22.918 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:22.919 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj290 datafile 2019-12-21T15:51:22.998 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:22.999 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj291 datafile 2019-12-21T15:51:23.082 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:23.082 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj292 datafile 2019-12-21T15:51:23.174 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:23.174 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj293 datafile 2019-12-21T15:51:23.217 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:23.217 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj294 datafile 2019-12-21T15:51:23.258 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:23.259 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj295 datafile 2019-12-21T15:51:23.310 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:23.310 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj296 datafile 2019-12-21T15:51:23.350 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:23.350 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj297 datafile 2019-12-21T15:51:23.408 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:23.408 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj298 datafile 2019-12-21T15:51:23.450 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:23.450 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj299 datafile 2019-12-21T15:51:23.493 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:23.494 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj300 datafile 2019-12-21T15:51:23.591 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:23.591 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj301 datafile 2019-12-21T15:51:23.675 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:23.675 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj302 datafile 2019-12-21T15:51:23.751 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:23.751 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj303 datafile 2019-12-21T15:51:23.792 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:23.792 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj304 datafile 2019-12-21T15:51:23.835 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:23.835 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj305 datafile 2019-12-21T15:51:23.926 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:23.926 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj306 datafile 2019-12-21T15:51:24.010 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:24.010 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj307 datafile 2019-12-21T15:51:24.059 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:24.059 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj308 datafile 2019-12-21T15:51:24.100 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:24.100 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj309 datafile 2019-12-21T15:51:24.185 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:24.185 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj310 datafile 2019-12-21T15:51:24.255 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:24.256 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj311 datafile 2019-12-21T15:51:24.335 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:24.335 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj312 datafile 2019-12-21T15:51:24.377 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:24.377 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj313 datafile 2019-12-21T15:51:24.461 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:24.461 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj314 datafile 2019-12-21T15:51:24.510 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:24.510 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj315 datafile 2019-12-21T15:51:24.556 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:24.556 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj316 datafile 2019-12-21T15:51:24.592 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:24.592 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj317 datafile 2019-12-21T15:51:24.736 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:24.737 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj318 datafile 2019-12-21T15:51:24.792 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:24.792 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj319 datafile 2019-12-21T15:51:24.895 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:24.896 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj320 datafile 2019-12-21T15:51:24.958 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:24.959 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj321 datafile 2019-12-21T15:51:25.009 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:25.009 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj322 datafile 2019-12-21T15:51:25.051 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:25.051 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj323 datafile 2019-12-21T15:51:25.129 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:25.129 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj324 datafile 2019-12-21T15:51:25.175 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:25.175 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj325 datafile 2019-12-21T15:51:25.219 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:25.219 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj326 datafile 2019-12-21T15:51:25.267 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:25.268 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj327 datafile 2019-12-21T15:51:25.354 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:25.355 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj328 datafile 2019-12-21T15:51:25.505 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:25.505 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj329 datafile 2019-12-21T15:51:25.564 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:25.564 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj330 datafile 2019-12-21T15:51:25.675 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:25.676 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj331 datafile 2019-12-21T15:51:25.717 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:25.717 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj332 datafile 2019-12-21T15:51:25.759 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:25.759 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj333 datafile 2019-12-21T15:51:25.840 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:25.840 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj334 datafile 2019-12-21T15:51:25.884 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:25.884 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj335 datafile 2019-12-21T15:51:25.926 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:25.926 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj336 datafile 2019-12-21T15:51:26.032 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:26.032 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj337 datafile 2019-12-21T15:51:26.076 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:26.076 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj338 datafile 2019-12-21T15:51:26.117 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:26.117 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj339 datafile 2019-12-21T15:51:26.190 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:26.190 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj340 datafile 2019-12-21T15:51:26.274 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:26.274 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj341 datafile 2019-12-21T15:51:26.318 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:26.318 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj342 datafile 2019-12-21T15:51:26.391 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:26.391 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj343 datafile 2019-12-21T15:51:26.434 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:26.434 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj344 datafile 2019-12-21T15:51:26.476 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:26.476 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj345 datafile 2019-12-21T15:51:26.526 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:26.526 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj346 datafile 2019-12-21T15:51:26.567 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:26.568 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj347 datafile 2019-12-21T15:51:26.719 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:26.719 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj348 datafile 2019-12-21T15:51:26.767 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:26.768 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj349 datafile 2019-12-21T15:51:26.859 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:26.859 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj350 datafile 2019-12-21T15:51:26.934 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:26.934 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj351 datafile 2019-12-21T15:51:26.984 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:26.984 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj352 datafile 2019-12-21T15:51:27.068 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:27.068 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj353 datafile 2019-12-21T15:51:27.111 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:27.111 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj354 datafile 2019-12-21T15:51:27.159 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:27.159 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj355 datafile 2019-12-21T15:51:27.209 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:27.210 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj356 datafile 2019-12-21T15:51:27.302 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:27.302 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj357 datafile 2019-12-21T15:51:27.394 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:27.394 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj358 datafile 2019-12-21T15:51:27.445 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:27.445 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj359 datafile 2019-12-21T15:51:27.536 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:27.536 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj360 datafile 2019-12-21T15:51:27.620 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:27.620 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj361 datafile 2019-12-21T15:51:27.670 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:27.670 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj362 datafile 2019-12-21T15:51:27.718 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:27.718 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj363 datafile 2019-12-21T15:51:27.784 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:27.784 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj364 datafile 2019-12-21T15:51:27.834 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:27.835 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj365 datafile 2019-12-21T15:51:27.876 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:27.876 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj366 datafile 2019-12-21T15:51:27.926 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:27.926 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj367 datafile 2019-12-21T15:51:27.967 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:27.967 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj368 datafile 2019-12-21T15:51:28.078 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:28.079 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj369 datafile 2019-12-21T15:51:28.170 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:28.171 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj370 datafile 2019-12-21T15:51:28.254 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:28.254 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj371 datafile 2019-12-21T15:51:28.348 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:28.348 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj372 datafile 2019-12-21T15:51:28.401 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:28.401 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj373 datafile 2019-12-21T15:51:28.525 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:28.525 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj374 datafile 2019-12-21T15:51:28.584 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:28.585 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj375 datafile 2019-12-21T15:51:28.691 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:28.691 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj376 datafile 2019-12-21T15:51:28.735 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:28.735 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj377 datafile 2019-12-21T15:51:28.785 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:28.785 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj378 datafile 2019-12-21T15:51:28.834 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:28.835 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj379 datafile 2019-12-21T15:51:28.885 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:28.885 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj380 datafile 2019-12-21T15:51:28.966 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:28.967 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj381 datafile 2019-12-21T15:51:29.017 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:29.018 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj382 datafile 2019-12-21T15:51:29.070 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:29.070 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj383 datafile 2019-12-21T15:51:29.184 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:29.184 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj384 datafile 2019-12-21T15:51:29.301 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:29.302 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj385 datafile 2019-12-21T15:51:29.360 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:29.360 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj386 datafile 2019-12-21T15:51:29.411 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:29.412 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj387 datafile 2019-12-21T15:51:29.461 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:29.462 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj388 datafile 2019-12-21T15:51:29.552 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:29.552 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj389 datafile 2019-12-21T15:51:29.603 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:29.604 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj390 datafile 2019-12-21T15:51:29.651 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:29.652 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj391 datafile 2019-12-21T15:51:29.743 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:29.743 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj392 datafile 2019-12-21T15:51:29.835 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:29.836 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj393 datafile 2019-12-21T15:51:29.892 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:29.893 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj394 datafile 2019-12-21T15:51:29.986 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:29.986 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj395 datafile 2019-12-21T15:51:30.035 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:30.035 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj396 datafile 2019-12-21T15:51:30.153 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:30.153 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj397 datafile 2019-12-21T15:51:30.253 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:30.253 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj398 datafile 2019-12-21T15:51:30.293 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:30.293 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj399 datafile 2019-12-21T15:51:30.354 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:30.354 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj400 datafile 2019-12-21T15:51:30.393 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:30.393 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj401 datafile 2019-12-21T15:51:30.479 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:30.479 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj402 datafile 2019-12-21T15:51:30.649 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:30.649 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj403 datafile 2019-12-21T15:51:30.780 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:30.780 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj404 datafile 2019-12-21T15:51:30.826 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:30.827 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj405 datafile 2019-12-21T15:51:30.876 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:30.877 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj406 datafile 2019-12-21T15:51:30.927 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:30.927 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj407 datafile 2019-12-21T15:51:30.976 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:30.977 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj408 datafile 2019-12-21T15:51:31.027 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:31.027 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj409 datafile 2019-12-21T15:51:31.114 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:31.114 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj410 datafile 2019-12-21T15:51:31.160 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:31.160 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj411 datafile 2019-12-21T15:51:31.256 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:31.256 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj412 datafile 2019-12-21T15:51:31.310 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:31.310 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj413 datafile 2019-12-21T15:51:31.398 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:31.398 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj414 datafile 2019-12-21T15:51:31.443 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:31.443 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj415 datafile 2019-12-21T15:51:31.531 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:31.532 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj416 datafile 2019-12-21T15:51:31.576 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:31.577 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj417 datafile 2019-12-21T15:51:31.629 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:31.629 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj418 datafile 2019-12-21T15:51:31.724 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:31.724 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj419 datafile 2019-12-21T15:51:31.777 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:31.777 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj420 datafile 2019-12-21T15:51:31.857 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:31.857 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj421 datafile 2019-12-21T15:51:31.958 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:31.958 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj422 datafile 2019-12-21T15:51:32.018 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:32.019 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj423 datafile 2019-12-21T15:51:32.072 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:32.072 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj424 datafile 2019-12-21T15:51:32.118 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:32.118 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj425 datafile 2019-12-21T15:51:32.160 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:32.160 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj426 datafile 2019-12-21T15:51:32.210 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:32.210 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj427 datafile 2019-12-21T15:51:32.251 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:32.252 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj428 datafile 2019-12-21T15:51:32.347 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:32.347 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj429 datafile 2019-12-21T15:51:32.434 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:32.434 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj430 datafile 2019-12-21T15:51:32.485 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:32.486 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj431 datafile 2019-12-21T15:51:32.535 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:32.535 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj432 datafile 2019-12-21T15:51:32.610 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:32.610 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj433 datafile 2019-12-21T15:51:32.679 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:32.680 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj434 datafile 2019-12-21T15:51:32.776 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:32.777 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj435 datafile 2019-12-21T15:51:32.860 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:32.860 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj436 datafile 2019-12-21T15:51:32.910 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:32.910 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj437 datafile 2019-12-21T15:51:33.002 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:33.002 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj438 datafile 2019-12-21T15:51:33.052 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:33.052 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj439 datafile 2019-12-21T15:51:33.136 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:33.136 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj440 datafile 2019-12-21T15:51:33.236 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:33.236 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj441 datafile 2019-12-21T15:51:33.287 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:33.287 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj442 datafile 2019-12-21T15:51:33.387 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:33.387 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj443 datafile 2019-12-21T15:51:33.452 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:33.453 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj444 datafile 2019-12-21T15:51:33.502 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:33.502 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj445 datafile 2019-12-21T15:51:33.587 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:33.587 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj446 datafile 2019-12-21T15:51:33.636 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:33.636 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj447 datafile 2019-12-21T15:51:33.677 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:33.677 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj448 datafile 2019-12-21T15:51:33.727 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:33.728 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj449 datafile 2019-12-21T15:51:33.778 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:33.778 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj450 datafile 2019-12-21T15:51:33.871 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:33.871 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj451 datafile 2019-12-21T15:51:33.911 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:33.911 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj452 datafile 2019-12-21T15:51:34.005 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:34.005 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj453 datafile 2019-12-21T15:51:34.106 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:34.107 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj454 datafile 2019-12-21T15:51:34.180 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:34.181 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj455 datafile 2019-12-21T15:51:34.227 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:34.227 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj456 datafile 2019-12-21T15:51:34.331 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:34.331 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj457 datafile 2019-12-21T15:51:34.386 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:34.386 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj458 datafile 2019-12-21T15:51:34.428 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:34.428 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj459 datafile 2019-12-21T15:51:34.478 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:34.478 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj460 datafile 2019-12-21T15:51:34.540 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:34.540 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj461 datafile 2019-12-21T15:51:34.594 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:34.595 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj462 datafile 2019-12-21T15:51:34.741 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:34.741 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj463 datafile 2019-12-21T15:51:34.785 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:34.786 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj464 datafile 2019-12-21T15:51:34.878 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:34.878 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj465 datafile 2019-12-21T15:51:34.936 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:34.937 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj466 datafile 2019-12-21T15:51:35.016 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:35.016 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj467 datafile 2019-12-21T15:51:35.070 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:35.070 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj468 datafile 2019-12-21T15:51:35.142 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:35.142 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj469 datafile 2019-12-21T15:51:35.196 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:35.196 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj470 datafile 2019-12-21T15:51:35.275 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:35.275 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj471 datafile 2019-12-21T15:51:35.367 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:35.368 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj472 datafile 2019-12-21T15:51:35.421 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:35.421 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj473 datafile 2019-12-21T15:51:35.539 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:35.539 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj474 datafile 2019-12-21T15:51:35.586 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:35.586 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj475 datafile 2019-12-21T15:51:35.651 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:35.651 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj476 datafile 2019-12-21T15:51:35.695 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:35.695 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj477 datafile 2019-12-21T15:51:35.777 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:35.777 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj478 datafile 2019-12-21T15:51:35.819 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:35.819 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj479 datafile 2019-12-21T15:51:35.870 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:35.870 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj480 datafile 2019-12-21T15:51:35.933 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:35.933 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj481 datafile 2019-12-21T15:51:36.018 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:36.019 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj482 datafile 2019-12-21T15:51:36.111 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:36.111 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj483 datafile 2019-12-21T15:51:36.215 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:36.216 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj484 datafile 2019-12-21T15:51:36.261 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:36.261 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj485 datafile 2019-12-21T15:51:36.319 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:36.320 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj486 datafile 2019-12-21T15:51:36.403 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:36.403 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj487 datafile 2019-12-21T15:51:36.453 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:36.454 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj488 datafile 2019-12-21T15:51:36.545 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:36.545 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj489 datafile 2019-12-21T15:51:36.595 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:36.595 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj490 datafile 2019-12-21T15:51:36.696 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:36.696 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj491 datafile 2019-12-21T15:51:36.745 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:36.745 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj492 datafile 2019-12-21T15:51:36.788 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:36.788 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj493 datafile 2019-12-21T15:51:36.888 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:36.888 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj494 datafile 2019-12-21T15:51:36.963 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:36.964 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj495 datafile 2019-12-21T15:51:37.014 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:37.014 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj496 datafile 2019-12-21T15:51:37.054 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:37.054 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj497 datafile 2019-12-21T15:51:37.095 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:37.096 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj498 datafile 2019-12-21T15:51:37.189 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:37.189 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj499 datafile 2019-12-21T15:51:37.235 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:37.235 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj500 datafile 2019-12-21T15:51:37.288 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:37.288 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj501 datafile 2019-12-21T15:51:37.337 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:37.338 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj502 datafile 2019-12-21T15:51:37.380 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:37.380 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj503 datafile 2019-12-21T15:51:37.481 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:37.481 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj504 datafile 2019-12-21T15:51:37.573 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:37.574 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj505 datafile 2019-12-21T15:51:37.674 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:37.674 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj506 datafile 2019-12-21T15:51:37.757 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:37.758 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj507 datafile 2019-12-21T15:51:37.812 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:37.812 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj508 datafile 2019-12-21T15:51:37.862 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:37.862 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj509 datafile 2019-12-21T15:51:37.913 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:37.913 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj510 datafile 2019-12-21T15:51:37.955 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:37.955 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj511 datafile 2019-12-21T15:51:37.995 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:37.995 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj512 datafile 2019-12-21T15:51:38.045 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:38.045 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj513 datafile 2019-12-21T15:51:38.096 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:38.096 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj514 datafile 2019-12-21T15:51:38.192 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:38.192 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj515 datafile 2019-12-21T15:51:38.266 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:38.267 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj516 datafile 2019-12-21T15:51:38.333 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:38.333 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj517 datafile 2019-12-21T15:51:38.392 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:38.392 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj518 datafile 2019-12-21T15:51:38.446 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:38.446 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj519 datafile 2019-12-21T15:51:38.487 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:38.487 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj520 datafile 2019-12-21T15:51:38.537 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:38.539 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj521 datafile 2019-12-21T15:51:38.659 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:38.659 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj522 datafile 2019-12-21T15:51:38.752 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:38.753 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj523 datafile 2019-12-21T15:51:38.835 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:38.835 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj524 datafile 2019-12-21T15:51:38.935 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:38.936 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj525 datafile 2019-12-21T15:51:39.061 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:39.061 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj526 datafile 2019-12-21T15:51:39.113 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:39.113 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj527 datafile 2019-12-21T15:51:39.153 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:39.154 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj528 datafile 2019-12-21T15:51:39.195 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:39.196 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj529 datafile 2019-12-21T15:51:39.320 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:39.320 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj530 datafile 2019-12-21T15:51:39.371 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:39.371 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj531 datafile 2019-12-21T15:51:39.412 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:39.412 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj532 datafile 2019-12-21T15:51:39.455 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:39.455 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj533 datafile 2019-12-21T15:51:39.497 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:39.497 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj534 datafile 2019-12-21T15:51:39.604 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:39.604 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj535 datafile 2019-12-21T15:51:39.654 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:39.654 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj536 datafile 2019-12-21T15:51:39.779 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:39.780 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj537 datafile 2019-12-21T15:51:39.818 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:39.818 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj538 datafile 2019-12-21T15:51:39.879 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:39.880 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj539 datafile 2019-12-21T15:51:39.963 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:39.963 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj540 datafile 2019-12-21T15:51:40.005 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:40.005 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj541 datafile 2019-12-21T15:51:40.045 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:40.045 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj542 datafile 2019-12-21T15:51:40.095 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:40.096 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj543 datafile 2019-12-21T15:51:40.138 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:40.138 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj544 datafile 2019-12-21T15:51:40.179 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:40.179 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj545 datafile 2019-12-21T15:51:40.264 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:40.264 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj546 datafile 2019-12-21T15:51:40.305 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:40.305 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj547 datafile 2019-12-21T15:51:40.398 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:40.398 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj548 datafile 2019-12-21T15:51:40.455 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:40.455 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj549 datafile 2019-12-21T15:51:40.540 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:40.540 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj550 datafile 2019-12-21T15:51:40.648 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:40.649 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj551 datafile 2019-12-21T15:51:40.688 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:40.689 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj552 datafile 2019-12-21T15:51:40.784 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:40.784 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj553 datafile 2019-12-21T15:51:40.830 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:40.831 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj554 datafile 2019-12-21T15:51:40.872 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:40.872 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj555 datafile 2019-12-21T15:51:40.950 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:40.950 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj556 datafile 2019-12-21T15:51:40.994 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:40.995 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj557 datafile 2019-12-21T15:51:41.075 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:41.075 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj558 datafile 2019-12-21T15:51:41.121 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:41.121 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj559 datafile 2019-12-21T15:51:41.172 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:41.172 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj560 datafile 2019-12-21T15:51:41.213 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:41.213 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj561 datafile 2019-12-21T15:51:41.255 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:41.256 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj562 datafile 2019-12-21T15:51:41.342 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:41.343 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj563 datafile 2019-12-21T15:51:41.400 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:41.401 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj564 datafile 2019-12-21T15:51:41.493 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:41.493 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj565 datafile 2019-12-21T15:51:41.576 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:41.577 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj566 datafile 2019-12-21T15:51:41.660 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:41.660 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj567 datafile 2019-12-21T15:51:41.705 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:41.706 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj568 datafile 2019-12-21T15:51:41.746 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:41.746 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj569 datafile 2019-12-21T15:51:41.848 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:41.848 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj570 datafile 2019-12-21T15:51:41.896 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:41.896 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj571 datafile 2019-12-21T15:51:41.937 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:41.938 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj572 datafile 2019-12-21T15:51:42.019 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:42.019 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj573 datafile 2019-12-21T15:51:42.103 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:42.103 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj574 datafile 2019-12-21T15:51:42.178 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:42.178 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj575 datafile 2019-12-21T15:51:42.222 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:42.222 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj576 datafile 2019-12-21T15:51:42.261 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:42.261 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj577 datafile 2019-12-21T15:51:42.337 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:42.337 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj578 datafile 2019-12-21T15:51:42.380 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:42.380 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj579 datafile 2019-12-21T15:51:42.422 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:42.423 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj580 datafile 2019-12-21T15:51:42.461 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:42.462 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj581 datafile 2019-12-21T15:51:42.505 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:42.506 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj582 datafile 2019-12-21T15:51:42.579 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:42.580 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj583 datafile 2019-12-21T15:51:42.679 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:42.679 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj584 datafile 2019-12-21T15:51:42.780 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:42.780 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj585 datafile 2019-12-21T15:51:42.818 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:42.818 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj586 datafile 2019-12-21T15:51:42.918 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:42.918 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj587 datafile 2019-12-21T15:51:42.955 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:42.956 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj588 datafile 2019-12-21T15:51:42.994 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:42.995 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj589 datafile 2019-12-21T15:51:43.056 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:43.058 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj590 datafile 2019-12-21T15:51:43.097 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:43.097 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj591 datafile 2019-12-21T15:51:43.137 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:43.138 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj592 datafile 2019-12-21T15:51:43.273 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:43.273 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj593 datafile 2019-12-21T15:51:43.357 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:43.357 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj594 datafile 2019-12-21T15:51:43.403 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:43.403 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj595 datafile 2019-12-21T15:51:43.447 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:43.448 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj596 datafile 2019-12-21T15:51:43.487 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:43.488 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj597 datafile 2019-12-21T15:51:43.529 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:43.530 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj598 datafile 2019-12-21T15:51:43.571 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:43.571 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj599 datafile 2019-12-21T15:51:43.657 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:43.657 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj600 datafile 2019-12-21T15:51:43.732 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:43.733 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj601 datafile 2019-12-21T15:51:43.773 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:43.773 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj602 datafile 2019-12-21T15:51:43.858 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:43.858 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj603 datafile 2019-12-21T15:51:43.903 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:43.903 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj604 datafile 2019-12-21T15:51:43.964 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:43.964 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj605 datafile 2019-12-21T15:51:44.006 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:44.006 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj606 datafile 2019-12-21T15:51:44.092 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:44.092 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj607 datafile 2019-12-21T15:51:44.131 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:44.131 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj608 datafile 2019-12-21T15:51:44.169 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:44.169 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj609 datafile 2019-12-21T15:51:44.242 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:44.242 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj610 datafile 2019-12-21T15:51:44.334 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:44.334 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj611 datafile 2019-12-21T15:51:44.393 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:44.393 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj612 datafile 2019-12-21T15:51:44.436 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:44.437 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj613 datafile 2019-12-21T15:51:44.473 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:44.473 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj614 datafile 2019-12-21T15:51:44.510 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:44.511 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj615 datafile 2019-12-21T15:51:44.665 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:44.666 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj616 datafile 2019-12-21T15:51:44.735 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:44.736 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj617 datafile 2019-12-21T15:51:44.773 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:44.774 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj618 datafile 2019-12-21T15:51:44.814 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:44.814 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj619 datafile 2019-12-21T15:51:44.920 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:44.920 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj620 datafile 2019-12-21T15:51:45.011 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:45.012 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj621 datafile 2019-12-21T15:51:45.071 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:45.071 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj622 datafile 2019-12-21T15:51:45.107 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:45.107 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj623 datafile 2019-12-21T15:51:45.144 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:45.145 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj624 datafile 2019-12-21T15:51:45.220 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:45.221 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj625 datafile 2019-12-21T15:51:45.261 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:45.262 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj626 datafile 2019-12-21T15:51:45.368 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:45.368 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj627 datafile 2019-12-21T15:51:45.446 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:45.447 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj628 datafile 2019-12-21T15:51:45.490 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:45.490 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj629 datafile 2019-12-21T15:51:45.564 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:45.564 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj630 datafile 2019-12-21T15:51:45.606 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:45.606 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj631 datafile 2019-12-21T15:51:45.645 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:45.645 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj632 datafile 2019-12-21T15:51:45.687 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:45.687 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj633 datafile 2019-12-21T15:51:45.772 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:45.773 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj634 datafile 2019-12-21T15:51:45.810 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:45.811 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj635 datafile 2019-12-21T15:51:45.853 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:45.853 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj636 datafile 2019-12-21T15:51:45.895 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:45.895 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj637 datafile 2019-12-21T15:51:46.006 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:46.006 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj638 datafile 2019-12-21T15:51:46.044 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:46.044 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj639 datafile 2019-12-21T15:51:46.131 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:46.132 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj640 datafile 2019-12-21T15:51:46.174 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:46.174 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj641 datafile 2019-12-21T15:51:46.211 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:46.212 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj642 datafile 2019-12-21T15:51:46.332 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:46.332 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj643 datafile 2019-12-21T15:51:46.369 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:46.369 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj644 datafile 2019-12-21T15:51:46.449 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:46.449 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj645 datafile 2019-12-21T15:51:46.495 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:46.495 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj646 datafile 2019-12-21T15:51:46.532 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:46.533 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj647 datafile 2019-12-21T15:51:46.616 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:46.616 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj648 datafile 2019-12-21T15:51:46.666 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:46.667 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj649 datafile 2019-12-21T15:51:46.768 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:46.769 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj650 datafile 2019-12-21T15:51:46.815 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:46.815 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj651 datafile 2019-12-21T15:51:46.864 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:46.865 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj652 datafile 2019-12-21T15:51:46.908 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:46.908 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj653 datafile 2019-12-21T15:51:46.944 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:46.944 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj654 datafile 2019-12-21T15:51:47.059 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:47.059 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj655 datafile 2019-12-21T15:51:47.106 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:47.107 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj656 datafile 2019-12-21T15:51:47.184 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:47.184 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj657 datafile 2019-12-21T15:51:47.268 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:47.268 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj658 datafile 2019-12-21T15:51:47.314 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:47.314 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj659 datafile 2019-12-21T15:51:47.349 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:47.350 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj660 datafile 2019-12-21T15:51:47.390 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:47.390 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj661 datafile 2019-12-21T15:51:47.510 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:47.511 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj662 datafile 2019-12-21T15:51:47.558 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:47.558 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj663 datafile 2019-12-21T15:51:47.599 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:47.600 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj664 datafile 2019-12-21T15:51:47.702 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:47.703 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj665 datafile 2019-12-21T15:51:47.740 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:47.740 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj666 datafile 2019-12-21T15:51:47.819 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:47.819 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj667 datafile 2019-12-21T15:51:47.895 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:47.895 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj668 datafile 2019-12-21T15:51:47.940 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:47.940 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj669 datafile 2019-12-21T15:51:47.978 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:47.978 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj670 datafile 2019-12-21T15:51:48.024 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:48.025 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj671 datafile 2019-12-21T15:51:48.066 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:48.066 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj672 datafile 2019-12-21T15:51:48.103 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:48.103 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj673 datafile 2019-12-21T15:51:48.148 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:48.149 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj674 datafile 2019-12-21T15:51:48.220 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:48.220 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj675 datafile 2019-12-21T15:51:48.312 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:48.312 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj676 datafile 2019-12-21T15:51:48.388 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:48.388 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj677 datafile 2019-12-21T15:51:48.433 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:48.434 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj678 datafile 2019-12-21T15:51:48.513 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:48.513 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj679 datafile 2019-12-21T15:51:48.557 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:48.557 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj680 datafile 2019-12-21T15:51:48.591 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:48.592 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj681 datafile 2019-12-21T15:51:48.651 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:48.651 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj682 datafile 2019-12-21T15:51:48.692 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:48.692 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj683 datafile 2019-12-21T15:51:48.772 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:48.772 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj684 datafile 2019-12-21T15:51:48.847 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:48.848 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj685 datafile 2019-12-21T15:51:48.890 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:48.890 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj686 datafile 2019-12-21T15:51:48.965 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:48.965 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj687 datafile 2019-12-21T15:51:49.065 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:49.065 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj688 datafile 2019-12-21T15:51:49.160 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:49.160 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj689 datafile 2019-12-21T15:51:49.200 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:49.200 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj690 datafile 2019-12-21T15:51:49.278 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:49.278 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj691 datafile 2019-12-21T15:51:49.315 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:49.315 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj692 datafile 2019-12-21T15:51:49.364 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:49.364 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj693 datafile 2019-12-21T15:51:49.402 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:49.403 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj694 datafile 2019-12-21T15:51:49.446 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:49.446 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj695 datafile 2019-12-21T15:51:49.529 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:49.530 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj696 datafile 2019-12-21T15:51:49.617 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:49.617 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj697 datafile 2019-12-21T15:51:49.700 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:49.701 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj698 datafile 2019-12-21T15:51:49.784 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:49.784 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj699 datafile 2019-12-21T15:51:49.824 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:49.824 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj700 datafile 2019-12-21T15:51:49.917 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:49.918 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj701 datafile 2019-12-21T15:51:49.971 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:49.971 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj702 datafile 2019-12-21T15:51:50.008 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:50.009 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj703 datafile 2019-12-21T15:51:50.049 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:50.049 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj704 datafile 2019-12-21T15:51:50.091 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:50.091 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj705 datafile 2019-12-21T15:51:50.168 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:50.168 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj706 datafile 2019-12-21T15:51:50.208 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:50.209 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj707 datafile 2019-12-21T15:51:50.254 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:50.254 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj708 datafile 2019-12-21T15:51:50.304 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:50.304 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj709 datafile 2019-12-21T15:51:50.402 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:50.402 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj710 datafile 2019-12-21T15:51:50.511 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:50.511 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj711 datafile 2019-12-21T15:51:50.603 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:50.603 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj712 datafile 2019-12-21T15:51:50.654 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:50.654 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj713 datafile 2019-12-21T15:51:50.699 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:50.699 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj714 datafile 2019-12-21T15:51:50.742 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:50.743 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj715 datafile 2019-12-21T15:51:50.829 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:50.829 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj716 datafile 2019-12-21T15:51:50.869 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:50.870 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj717 datafile 2019-12-21T15:51:50.954 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:50.954 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj718 datafile 2019-12-21T15:51:51.012 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:51.012 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj719 datafile 2019-12-21T15:51:51.265 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:51.265 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj720 datafile 2019-12-21T15:51:51.430 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:51.430 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj721 datafile 2019-12-21T15:51:51.789 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:51.789 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj722 datafile 2019-12-21T15:51:51.834 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:51.835 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj723 datafile 2019-12-21T15:51:51.878 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:51.879 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj724 datafile 2019-12-21T15:51:51.915 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:51.915 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj725 datafile 2019-12-21T15:51:52.023 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:52.024 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj726 datafile 2019-12-21T15:51:52.059 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:52.059 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj727 datafile 2019-12-21T15:51:52.103 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:52.108 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj728 datafile 2019-12-21T15:51:52.198 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:52.198 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj729 datafile 2019-12-21T15:51:52.241 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:52.241 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj730 datafile 2019-12-21T15:51:52.279 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:52.279 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj731 datafile 2019-12-21T15:51:52.329 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:52.329 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj732 datafile 2019-12-21T15:51:52.374 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:52.374 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj733 datafile 2019-12-21T15:51:52.424 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:52.424 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj734 datafile 2019-12-21T15:51:52.466 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:52.466 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj735 datafile 2019-12-21T15:51:52.504 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:52.504 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj736 datafile 2019-12-21T15:51:52.590 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:52.590 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj737 datafile 2019-12-21T15:51:52.651 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:52.651 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj738 datafile 2019-12-21T15:51:52.735 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:52.735 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj739 datafile 2019-12-21T15:51:52.807 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:52.808 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj740 datafile 2019-12-21T15:51:52.891 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:52.891 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj741 datafile 2019-12-21T15:51:52.933 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:52.933 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj742 datafile 2019-12-21T15:51:53.017 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:53.017 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj743 datafile 2019-12-21T15:51:53.060 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:53.061 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj744 datafile 2019-12-21T15:51:53.124 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:53.125 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj745 datafile 2019-12-21T15:51:53.166 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:53.166 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj746 datafile 2019-12-21T15:51:53.204 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:53.205 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj747 datafile 2019-12-21T15:51:53.241 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:53.241 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj748 datafile 2019-12-21T15:51:53.325 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:53.325 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj749 datafile 2019-12-21T15:51:53.409 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:53.409 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj750 datafile 2019-12-21T15:51:53.493 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:53.493 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj751 datafile 2019-12-21T15:51:53.569 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:53.569 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj752 datafile 2019-12-21T15:51:53.618 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:53.618 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj753 datafile 2019-12-21T15:51:53.702 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:53.702 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj754 datafile 2019-12-21T15:51:53.749 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:53.750 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj755 datafile 2019-12-21T15:51:53.794 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:53.794 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj756 datafile 2019-12-21T15:51:53.838 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:53.838 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj757 datafile 2019-12-21T15:51:53.874 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:53.874 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj758 datafile 2019-12-21T15:51:53.953 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:53.953 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj759 datafile 2019-12-21T15:51:54.045 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:54.045 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj760 datafile 2019-12-21T15:51:54.129 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:54.129 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj761 datafile 2019-12-21T15:51:54.175 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:54.175 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj762 datafile 2019-12-21T15:51:54.213 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:54.213 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj763 datafile 2019-12-21T15:51:54.261 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:54.261 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj764 datafile 2019-12-21T15:51:54.320 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:54.321 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj765 datafile 2019-12-21T15:51:54.361 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:54.361 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj766 datafile 2019-12-21T15:51:54.400 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:54.400 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj767 datafile 2019-12-21T15:51:54.522 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:54.522 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj768 datafile 2019-12-21T15:51:54.569 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:54.570 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj769 datafile 2019-12-21T15:51:54.688 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:54.689 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj770 datafile 2019-12-21T15:51:54.764 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:54.764 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj771 datafile 2019-12-21T15:51:54.808 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:54.808 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj772 datafile 2019-12-21T15:51:54.890 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:54.890 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj773 datafile 2019-12-21T15:51:55.015 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:55.015 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj774 datafile 2019-12-21T15:51:55.060 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:55.061 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj775 datafile 2019-12-21T15:51:55.105 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:55.105 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj776 datafile 2019-12-21T15:51:55.144 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:55.144 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj777 datafile 2019-12-21T15:51:55.183 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:55.183 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj778 datafile 2019-12-21T15:51:55.282 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:55.282 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj779 datafile 2019-12-21T15:51:55.327 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:55.328 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj780 datafile 2019-12-21T15:51:55.366 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:55.366 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj781 datafile 2019-12-21T15:51:55.449 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:55.449 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj782 datafile 2019-12-21T15:51:55.541 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:55.541 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj783 datafile 2019-12-21T15:51:55.586 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:55.586 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj784 datafile 2019-12-21T15:51:55.666 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:55.667 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj785 datafile 2019-12-21T15:51:55.711 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:55.711 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj786 datafile 2019-12-21T15:51:55.800 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:55.800 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj787 datafile 2019-12-21T15:51:55.846 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:55.847 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj788 datafile 2019-12-21T15:51:55.886 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:55.886 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj789 datafile 2019-12-21T15:51:55.936 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:55.936 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj790 datafile 2019-12-21T15:51:56.034 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:56.034 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj791 datafile 2019-12-21T15:51:56.117 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:56.118 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj792 datafile 2019-12-21T15:51:56.163 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:56.163 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj793 datafile 2019-12-21T15:51:56.208 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:56.208 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj794 datafile 2019-12-21T15:51:56.244 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:56.244 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj795 datafile 2019-12-21T15:51:56.286 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:56.286 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj796 datafile 2019-12-21T15:51:56.327 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:56.328 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj797 datafile 2019-12-21T15:51:56.372 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:56.372 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj798 datafile 2019-12-21T15:51:56.485 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:56.485 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj799 datafile 2019-12-21T15:51:56.528 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:56.528 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj800 datafile 2019-12-21T15:51:56.643 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:56.643 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj801 datafile 2019-12-21T15:51:56.694 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:56.695 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj802 datafile 2019-12-21T15:51:56.777 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:56.778 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj803 datafile 2019-12-21T15:51:56.816 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:56.817 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj804 datafile 2019-12-21T15:51:56.861 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:56.861 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj805 datafile 2019-12-21T15:51:56.907 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:56.907 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj806 datafile 2019-12-21T15:51:56.986 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:56.986 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj807 datafile 2019-12-21T15:51:57.061 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:57.062 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj808 datafile 2019-12-21T15:51:57.108 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:57.109 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj809 datafile 2019-12-21T15:51:57.150 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:57.150 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj810 datafile 2019-12-21T15:51:57.216 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:57.216 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj811 datafile 2019-12-21T15:51:57.258 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:57.259 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj812 datafile 2019-12-21T15:51:57.303 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:57.303 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj813 datafile 2019-12-21T15:51:57.421 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:57.421 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj814 datafile 2019-12-21T15:51:57.459 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:57.459 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj815 datafile 2019-12-21T15:51:57.565 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:57.565 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj816 datafile 2019-12-21T15:51:57.603 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:57.603 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj817 datafile 2019-12-21T15:51:57.644 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:57.645 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj818 datafile 2019-12-21T15:51:57.687 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:57.687 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj819 datafile 2019-12-21T15:51:57.728 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:57.728 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj820 datafile 2019-12-21T15:51:57.767 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:57.767 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj821 datafile 2019-12-21T15:51:57.812 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:57.812 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj822 datafile 2019-12-21T15:51:57.897 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:57.897 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj823 datafile 2019-12-21T15:51:57.980 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:57.981 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj824 datafile 2019-12-21T15:51:58.081 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:58.081 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj825 datafile 2019-12-21T15:51:58.173 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:58.173 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj826 datafile 2019-12-21T15:51:58.211 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:58.211 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj827 datafile 2019-12-21T15:51:58.253 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:58.253 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj828 datafile 2019-12-21T15:51:58.288 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:58.289 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj829 datafile 2019-12-21T15:51:58.328 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:58.329 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj830 datafile 2019-12-21T15:51:58.415 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:58.415 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj831 datafile 2019-12-21T15:51:58.467 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:58.467 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj832 datafile 2019-12-21T15:51:58.557 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:58.557 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj833 datafile 2019-12-21T15:51:58.601 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:58.601 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj834 datafile 2019-12-21T15:51:58.708 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:58.708 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj835 datafile 2019-12-21T15:51:58.757 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:58.758 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj836 datafile 2019-12-21T15:51:58.841 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:58.842 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj837 datafile 2019-12-21T15:51:58.926 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:58.926 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj838 datafile 2019-12-21T15:51:58.967 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:58.967 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj839 datafile 2019-12-21T15:51:59.009 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:59.009 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj840 datafile 2019-12-21T15:51:59.049 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:59.049 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj841 datafile 2019-12-21T15:51:59.091 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:59.091 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj842 datafile 2019-12-21T15:51:59.136 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:59.137 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj843 datafile 2019-12-21T15:51:59.209 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:59.209 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj844 datafile 2019-12-21T15:51:59.318 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:59.318 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj845 datafile 2019-12-21T15:51:59.405 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:59.406 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj846 datafile 2019-12-21T15:51:59.467 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:59.467 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj847 datafile 2019-12-21T15:51:59.552 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:59.552 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj848 datafile 2019-12-21T15:51:59.600 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:59.601 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj849 datafile 2019-12-21T15:51:59.645 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:59.645 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj850 datafile 2019-12-21T15:51:59.682 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:59.683 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj851 datafile 2019-12-21T15:51:59.744 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:59.744 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj852 datafile 2019-12-21T15:51:59.920 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:59.920 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj853 datafile 2019-12-21T15:51:59.959 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:51:59.959 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj854 datafile 2019-12-21T15:52:00.001 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:52:00.001 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj855 datafile 2019-12-21T15:52:00.062 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:52:00.062 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj856 datafile 2019-12-21T15:52:00.187 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:52:00.187 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj857 datafile 2019-12-21T15:52:00.234 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:52:00.234 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj858 datafile 2019-12-21T15:52:00.272 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:52:00.272 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj859 datafile 2019-12-21T15:52:00.371 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:52:00.371 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj860 datafile 2019-12-21T15:52:00.455 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:52:00.455 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj861 datafile 2019-12-21T15:52:00.501 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:52:00.501 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj862 datafile 2019-12-21T15:52:00.542 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:52:00.543 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj863 datafile 2019-12-21T15:52:00.584 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:52:00.584 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj864 datafile 2019-12-21T15:52:00.651 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:52:00.651 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj865 datafile 2019-12-21T15:52:00.697 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:52:00.697 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj866 datafile 2019-12-21T15:52:00.742 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:52:00.743 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj867 datafile 2019-12-21T15:52:00.861 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:52:00.862 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj868 datafile 2019-12-21T15:52:00.947 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:52:00.947 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj869 datafile 2019-12-21T15:52:00.992 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:52:00.993 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj870 datafile 2019-12-21T15:52:01.090 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:52:01.090 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj871 datafile 2019-12-21T15:52:01.131 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:52:01.131 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj872 datafile 2019-12-21T15:52:01.215 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:52:01.215 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj873 datafile 2019-12-21T15:52:01.267 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:52:01.268 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj874 datafile 2019-12-21T15:52:01.309 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:52:01.309 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj875 datafile 2019-12-21T15:52:01.346 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:52:01.346 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj876 datafile 2019-12-21T15:52:01.393 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:52:01.393 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj877 datafile 2019-12-21T15:52:01.443 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:52:01.443 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj878 datafile 2019-12-21T15:52:01.484 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:52:01.484 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj879 datafile 2019-12-21T15:52:01.526 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:52:01.526 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj880 datafile 2019-12-21T15:52:01.584 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:52:01.584 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj881 datafile 2019-12-21T15:52:01.665 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:52:01.666 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj882 datafile 2019-12-21T15:52:01.758 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:52:01.758 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj883 datafile 2019-12-21T15:52:01.849 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:52:01.850 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj884 datafile 2019-12-21T15:52:01.893 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:52:01.893 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj885 datafile 2019-12-21T15:52:01.966 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:52:01.967 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj886 datafile 2019-12-21T15:52:02.009 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:52:02.009 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj887 datafile 2019-12-21T15:52:02.051 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:52:02.051 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj888 datafile 2019-12-21T15:52:02.094 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:52:02.094 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj889 datafile 2019-12-21T15:52:02.175 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:52:02.176 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj890 datafile 2019-12-21T15:52:02.213 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:52:02.213 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj891 datafile 2019-12-21T15:52:02.415 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:52:02.415 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj892 datafile 2019-12-21T15:52:02.476 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:52:02.476 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj893 datafile 2019-12-21T15:52:02.560 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:52:02.560 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj894 datafile 2019-12-21T15:52:02.601 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:52:02.601 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj895 datafile 2019-12-21T15:52:02.669 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:52:02.669 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj896 datafile 2019-12-21T15:52:02.718 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:52:02.718 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj897 datafile 2019-12-21T15:52:02.759 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:52:02.760 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj898 datafile 2019-12-21T15:52:02.836 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:52:02.836 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj899 datafile 2019-12-21T15:52:02.884 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:52:02.885 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj900 datafile 2019-12-21T15:52:02.994 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:52:02.995 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj901 datafile 2019-12-21T15:52:03.086 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:52:03.087 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj902 datafile 2019-12-21T15:52:03.131 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:52:03.131 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj903 datafile 2019-12-21T15:52:03.172 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:52:03.173 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj904 datafile 2019-12-21T15:52:03.218 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:52:03.218 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj905 datafile 2019-12-21T15:52:03.295 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:52:03.296 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj906 datafile 2019-12-21T15:52:03.371 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:52:03.371 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj907 datafile 2019-12-21T15:52:03.418 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:52:03.418 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj908 datafile 2019-12-21T15:52:03.464 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:52:03.465 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj909 datafile 2019-12-21T15:52:03.555 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:52:03.555 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj910 datafile 2019-12-21T15:52:03.638 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:52:03.639 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj911 datafile 2019-12-21T15:52:03.689 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:52:03.689 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj912 datafile 2019-12-21T15:52:03.733 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:52:03.733 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj913 datafile 2019-12-21T15:52:03.781 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:52:03.781 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj914 datafile 2019-12-21T15:52:03.826 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:52:03.827 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj915 datafile 2019-12-21T15:52:03.913 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:52:03.914 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj916 datafile 2019-12-21T15:52:03.958 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:52:03.959 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj917 datafile 2019-12-21T15:52:04.056 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:52:04.056 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj918 datafile 2019-12-21T15:52:04.097 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:52:04.098 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj919 datafile 2019-12-21T15:52:04.150 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:52:04.150 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj920 datafile 2019-12-21T15:52:04.231 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:52:04.232 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj921 datafile 2019-12-21T15:52:04.285 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:52:04.285 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj922 datafile 2019-12-21T15:52:04.325 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:52:04.325 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj923 datafile 2019-12-21T15:52:04.423 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:52:04.424 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj924 datafile 2019-12-21T15:52:04.464 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:52:04.464 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj925 datafile 2019-12-21T15:52:04.506 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:52:04.506 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj926 datafile 2019-12-21T15:52:04.590 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:52:04.591 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj927 datafile 2019-12-21T15:52:04.660 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:52:04.660 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj928 datafile 2019-12-21T15:52:04.698 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:52:04.698 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj929 datafile 2019-12-21T15:52:04.774 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:52:04.775 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj930 datafile 2019-12-21T15:52:04.814 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:52:04.815 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj931 datafile 2019-12-21T15:52:04.858 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:52:04.859 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj932 datafile 2019-12-21T15:52:04.942 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:52:04.942 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj933 datafile 2019-12-21T15:52:05.025 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:52:05.025 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj934 datafile 2019-12-21T15:52:05.102 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:52:05.102 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj935 datafile 2019-12-21T15:52:05.176 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:52:05.176 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj936 datafile 2019-12-21T15:52:05.217 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:52:05.217 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj937 datafile 2019-12-21T15:52:05.335 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:52:05.335 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj938 datafile 2019-12-21T15:52:05.385 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:52:05.385 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj939 datafile 2019-12-21T15:52:05.452 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:52:05.452 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj940 datafile 2019-12-21T15:52:05.535 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:52:05.535 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj941 datafile 2019-12-21T15:52:05.585 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:52:05.585 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj942 datafile 2019-12-21T15:52:05.642 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:52:05.643 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj943 datafile 2019-12-21T15:52:05.694 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:52:05.694 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj944 datafile 2019-12-21T15:52:05.740 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:52:05.740 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj945 datafile 2019-12-21T15:52:05.794 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:52:05.794 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj946 datafile 2019-12-21T15:52:05.878 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:52:05.878 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj947 datafile 2019-12-21T15:52:05.924 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:52:05.924 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj948 datafile 2019-12-21T15:52:06.003 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:52:06.003 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj949 datafile 2019-12-21T15:52:06.070 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:52:06.070 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj950 datafile 2019-12-21T15:52:06.154 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:52:06.154 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj951 datafile 2019-12-21T15:52:06.237 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:52:06.238 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj952 datafile 2019-12-21T15:52:06.346 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:52:06.346 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj953 datafile 2019-12-21T15:52:06.401 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:52:06.401 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj954 datafile 2019-12-21T15:52:06.443 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:52:06.443 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj955 datafile 2019-12-21T15:52:06.498 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:52:06.499 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj956 datafile 2019-12-21T15:52:06.588 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:52:06.588 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj957 datafile 2019-12-21T15:52:06.649 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:52:06.649 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj958 datafile 2019-12-21T15:52:06.694 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:52:06.694 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj959 datafile 2019-12-21T15:52:06.772 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:52:06.773 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj960 datafile 2019-12-21T15:52:06.857 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:52:06.915 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj961 datafile 2019-12-21T15:52:06.916 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:52:06.916 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj962 datafile 2019-12-21T15:52:06.940 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:52:06.940 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj963 datafile 2019-12-21T15:52:07.006 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:52:07.006 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj964 datafile 2019-12-21T15:52:07.049 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:52:07.049 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj965 datafile 2019-12-21T15:52:07.188 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:52:07.189 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj966 datafile 2019-12-21T15:52:07.242 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:52:07.242 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj967 datafile 2019-12-21T15:52:07.324 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:52:07.324 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj968 datafile 2019-12-21T15:52:07.407 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:52:07.408 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj969 datafile 2019-12-21T15:52:07.449 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:52:07.449 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj970 datafile 2019-12-21T15:52:07.500 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:52:07.500 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj971 datafile 2019-12-21T15:52:07.550 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:52:07.550 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj972 datafile 2019-12-21T15:52:07.607 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:52:07.608 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj973 datafile 2019-12-21T15:52:07.658 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:52:07.659 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj974 datafile 2019-12-21T15:52:07.741 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:52:07.742 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj975 datafile 2019-12-21T15:52:07.792 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:52:07.792 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj976 datafile 2019-12-21T15:52:07.842 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:52:07.842 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj977 datafile 2019-12-21T15:52:07.975 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:52:07.976 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj978 datafile 2019-12-21T15:52:08.068 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:52:08.068 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj979 datafile 2019-12-21T15:52:08.117 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:52:08.117 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj980 datafile 2019-12-21T15:52:08.158 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:52:08.158 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj981 datafile 2019-12-21T15:52:08.201 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:52:08.201 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj982 datafile 2019-12-21T15:52:08.293 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:52:08.293 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj983 datafile 2019-12-21T15:52:08.334 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:52:08.334 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj984 datafile 2019-12-21T15:52:08.384 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:52:08.384 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj985 datafile 2019-12-21T15:52:08.443 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:52:08.443 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj986 datafile 2019-12-21T15:52:08.527 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:52:08.527 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj987 datafile 2019-12-21T15:52:08.669 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:52:08.669 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj988 datafile 2019-12-21T15:52:08.717 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:52:08.717 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj989 datafile 2019-12-21T15:52:08.759 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:52:08.760 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj990 datafile 2019-12-21T15:52:08.845 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:52:08.845 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj991 datafile 2019-12-21T15:52:08.892 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:52:08.892 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj992 datafile 2019-12-21T15:52:08.942 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:52:08.943 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj993 datafile 2019-12-21T15:52:09.037 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:52:09.037 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj994 datafile 2019-12-21T15:52:09.083 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:52:09.083 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj995 datafile 2019-12-21T15:52:09.179 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:52:09.180 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj996 datafile 2019-12-21T15:52:09.313 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:52:09.313 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj997 datafile 2019-12-21T15:52:09.367 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:52:09.367 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj998 datafile 2019-12-21T15:52:09.463 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:52:09.464 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj999 datafile 2019-12-21T15:52:09.555 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:74: TEST_recover_unexpected: for j in $(seq 1 $OBJS) 2019-12-21T15:52:09.556 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:76: TEST_recover_unexpected: rados -p test3 put obj1000 datafile 2019-12-21T15:52:09.609 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:79: TEST_recover_unexpected: rm datafile 2019-12-21T15:52:09.609 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:81: TEST_recover_unexpected: ceph osd set noscrub 2019-12-21T15:52:09.974 INFO:tasks.workunit.client.0.smithi172.stderr:noscrub is set 2019-12-21T15:52:09.983 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:82: TEST_recover_unexpected: ceph osd set nodeep-scrub 2019-12-21T15:52:10.450 INFO:tasks.workunit.client.0.smithi172.stderr:nodeep-scrub is set 2019-12-21T15:52:10.460 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:84: TEST_recover_unexpected: ceph pg dump pgs --format=json-pretty 2019-12-21T15:52:10.460 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:84: TEST_recover_unexpected: jq '.pg_stats[].pgid' 2019-12-21T15:52:10.691 INFO:tasks.workunit.client.0.smithi172.stderr:dumped pgs 2019-12-21T15:52:10.699 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:84: TEST_recover_unexpected: for qpg in $(ceph pg dump pgs --format=json-pretty | jq '.pg_stats[].pgid') 2019-12-21T15:52:10.700 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: ceph pg dump pgs --format=json 2019-12-21T15:52:10.700 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: jq '.pg_stats[] | select(.pgid == "3.d") | .acting_primary' 2019-12-21T15:52:10.926 INFO:tasks.workunit.client.0.smithi172.stderr:dumped pgs 2019-12-21T15:52:10.934 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: primary=3 2019-12-21T15:52:10.935 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:87: TEST_recover_unexpected: eval 'pg="3.d"' 2019-12-21T15:52:10.935 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:87: TEST_recover_unexpected: pg=3.d 2019-12-21T15:52:10.935 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: ceph tell 3.d scrub 2019-12-21T15:52:11.048 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:52:11.048 INFO:tasks.workunit.client.0.smithi172.stdout: "deep": false, 2019-12-21T15:52:11.048 INFO:tasks.workunit.client.0.smithi172.stdout: "stamp": "2019-12-14T15:50:31.047070+0000" 2019-12-21T15:52:11.048 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:52:11.056 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:84: TEST_recover_unexpected: for qpg in $(ceph pg dump pgs --format=json-pretty | jq '.pg_stats[].pgid') 2019-12-21T15:52:11.057 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: ceph pg dump pgs --format=json 2019-12-21T15:52:11.057 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: jq '.pg_stats[] | select(.pgid == "1.f") | .acting_primary' 2019-12-21T15:52:11.284 INFO:tasks.workunit.client.0.smithi172.stderr:dumped pgs 2019-12-21T15:52:11.293 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: primary=3 2019-12-21T15:52:11.293 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:87: TEST_recover_unexpected: eval 'pg="1.f"' 2019-12-21T15:52:11.294 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:87: TEST_recover_unexpected: pg=1.f 2019-12-21T15:52:11.294 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: ceph tell 1.f scrub 2019-12-21T15:52:11.407 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:52:11.408 INFO:tasks.workunit.client.0.smithi172.stdout: "deep": false, 2019-12-21T15:52:11.408 INFO:tasks.workunit.client.0.smithi172.stdout: "stamp": "2019-12-14T15:50:31.406387+0000" 2019-12-21T15:52:11.408 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:52:11.416 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:84: TEST_recover_unexpected: for qpg in $(ceph pg dump pgs --format=json-pretty | jq '.pg_stats[].pgid') 2019-12-21T15:52:11.417 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: ceph pg dump pgs --format=json 2019-12-21T15:52:11.417 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: jq '.pg_stats[] | select(.pgid == "2.c") | .acting_primary' 2019-12-21T15:52:11.643 INFO:tasks.workunit.client.0.smithi172.stderr:dumped pgs 2019-12-21T15:52:11.652 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: primary=2 2019-12-21T15:52:11.652 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:87: TEST_recover_unexpected: eval 'pg="2.c"' 2019-12-21T15:52:11.652 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:87: TEST_recover_unexpected: pg=2.c 2019-12-21T15:52:11.652 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: ceph tell 2.c scrub 2019-12-21T15:52:11.767 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:52:11.767 INFO:tasks.workunit.client.0.smithi172.stdout: "deep": false, 2019-12-21T15:52:11.767 INFO:tasks.workunit.client.0.smithi172.stdout: "stamp": "2019-12-14T15:50:31.765715+0000" 2019-12-21T15:52:11.767 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:52:11.775 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:84: TEST_recover_unexpected: for qpg in $(ceph pg dump pgs --format=json-pretty | jq '.pg_stats[].pgid') 2019-12-21T15:52:11.776 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: ceph pg dump pgs --format=json 2019-12-21T15:52:11.776 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: jq '.pg_stats[] | select(.pgid == "3.c") | .acting_primary' 2019-12-21T15:52:12.000 INFO:tasks.workunit.client.0.smithi172.stderr:dumped pgs 2019-12-21T15:52:12.009 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: primary=5 2019-12-21T15:52:12.009 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:87: TEST_recover_unexpected: eval 'pg="3.c"' 2019-12-21T15:52:12.009 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:87: TEST_recover_unexpected: pg=3.c 2019-12-21T15:52:12.009 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: ceph tell 3.c scrub 2019-12-21T15:52:12.123 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:52:12.124 INFO:tasks.workunit.client.0.smithi172.stdout: "deep": false, 2019-12-21T15:52:12.124 INFO:tasks.workunit.client.0.smithi172.stdout: "stamp": "2019-12-14T15:50:32.121892+0000" 2019-12-21T15:52:12.124 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:52:12.131 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:84: TEST_recover_unexpected: for qpg in $(ceph pg dump pgs --format=json-pretty | jq '.pg_stats[].pgid') 2019-12-21T15:52:12.132 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: ceph pg dump pgs --format=json 2019-12-21T15:52:12.133 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: jq '.pg_stats[] | select(.pgid == "1.e") | .acting_primary' 2019-12-21T15:52:12.356 INFO:tasks.workunit.client.0.smithi172.stderr:dumped pgs 2019-12-21T15:52:12.366 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: primary=1 2019-12-21T15:52:12.366 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:87: TEST_recover_unexpected: eval 'pg="1.e"' 2019-12-21T15:52:12.366 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:87: TEST_recover_unexpected: pg=1.e 2019-12-21T15:52:12.366 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: ceph tell 1.e scrub 2019-12-21T15:52:12.479 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:52:12.479 INFO:tasks.workunit.client.0.smithi172.stdout: "deep": false, 2019-12-21T15:52:12.479 INFO:tasks.workunit.client.0.smithi172.stdout: "stamp": "2019-12-14T15:50:32.477832+0000" 2019-12-21T15:52:12.479 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:52:12.487 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:84: TEST_recover_unexpected: for qpg in $(ceph pg dump pgs --format=json-pretty | jq '.pg_stats[].pgid') 2019-12-21T15:52:12.488 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: ceph pg dump pgs --format=json 2019-12-21T15:52:12.488 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: jq '.pg_stats[] | select(.pgid == "2.d") | .acting_primary' 2019-12-21T15:52:12.714 INFO:tasks.workunit.client.0.smithi172.stderr:dumped pgs 2019-12-21T15:52:12.722 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: primary=5 2019-12-21T15:52:12.723 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:87: TEST_recover_unexpected: eval 'pg="2.d"' 2019-12-21T15:52:12.723 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:87: TEST_recover_unexpected: pg=2.d 2019-12-21T15:52:12.723 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: ceph tell 2.d scrub 2019-12-21T15:52:12.836 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:52:12.836 INFO:tasks.workunit.client.0.smithi172.stdout: "deep": false, 2019-12-21T15:52:12.837 INFO:tasks.workunit.client.0.smithi172.stdout: "stamp": "2019-12-14T15:50:32.835272+0000" 2019-12-21T15:52:12.837 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:52:12.845 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:84: TEST_recover_unexpected: for qpg in $(ceph pg dump pgs --format=json-pretty | jq '.pg_stats[].pgid') 2019-12-21T15:52:12.846 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: ceph pg dump pgs --format=json 2019-12-21T15:52:12.846 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: jq '.pg_stats[] | select(.pgid == "3.f") | .acting_primary' 2019-12-21T15:52:13.071 INFO:tasks.workunit.client.0.smithi172.stderr:dumped pgs 2019-12-21T15:52:13.080 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: primary=0 2019-12-21T15:52:13.080 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:87: TEST_recover_unexpected: eval 'pg="3.f"' 2019-12-21T15:52:13.080 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:87: TEST_recover_unexpected: pg=3.f 2019-12-21T15:52:13.081 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: ceph tell 3.f scrub 2019-12-21T15:52:13.193 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:52:13.194 INFO:tasks.workunit.client.0.smithi172.stdout: "deep": false, 2019-12-21T15:52:13.194 INFO:tasks.workunit.client.0.smithi172.stdout: "stamp": "2019-12-14T15:50:33.192542+0000" 2019-12-21T15:52:13.194 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:52:13.202 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:84: TEST_recover_unexpected: for qpg in $(ceph pg dump pgs --format=json-pretty | jq '.pg_stats[].pgid') 2019-12-21T15:52:13.203 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: ceph pg dump pgs --format=json 2019-12-21T15:52:13.203 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: jq '.pg_stats[] | select(.pgid == "1.d") | .acting_primary' 2019-12-21T15:52:13.429 INFO:tasks.workunit.client.0.smithi172.stderr:dumped pgs 2019-12-21T15:52:13.438 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: primary=2 2019-12-21T15:52:13.438 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:87: TEST_recover_unexpected: eval 'pg="1.d"' 2019-12-21T15:52:13.438 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:87: TEST_recover_unexpected: pg=1.d 2019-12-21T15:52:13.438 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: ceph tell 1.d scrub 2019-12-21T15:52:13.551 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:52:13.551 INFO:tasks.workunit.client.0.smithi172.stdout: "deep": false, 2019-12-21T15:52:13.552 INFO:tasks.workunit.client.0.smithi172.stdout: "stamp": "2019-12-14T15:50:33.550430+0000" 2019-12-21T15:52:13.552 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:52:13.560 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:84: TEST_recover_unexpected: for qpg in $(ceph pg dump pgs --format=json-pretty | jq '.pg_stats[].pgid') 2019-12-21T15:52:13.561 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: ceph pg dump pgs --format=json 2019-12-21T15:52:13.561 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: jq '.pg_stats[] | select(.pgid == "2.e") | .acting_primary' 2019-12-21T15:52:13.785 INFO:tasks.workunit.client.0.smithi172.stderr:dumped pgs 2019-12-21T15:52:13.793 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: primary=2 2019-12-21T15:52:13.794 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:87: TEST_recover_unexpected: eval 'pg="2.e"' 2019-12-21T15:52:13.794 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:87: TEST_recover_unexpected: pg=2.e 2019-12-21T15:52:13.794 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: ceph tell 2.e scrub 2019-12-21T15:52:13.907 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:52:13.907 INFO:tasks.workunit.client.0.smithi172.stdout: "deep": false, 2019-12-21T15:52:13.907 INFO:tasks.workunit.client.0.smithi172.stdout: "stamp": "2019-12-14T15:50:33.906226+0000" 2019-12-21T15:52:13.908 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:52:13.916 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:84: TEST_recover_unexpected: for qpg in $(ceph pg dump pgs --format=json-pretty | jq '.pg_stats[].pgid') 2019-12-21T15:52:13.917 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: ceph pg dump pgs --format=json 2019-12-21T15:52:13.917 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: jq '.pg_stats[] | select(.pgid == "3.e") | .acting_primary' 2019-12-21T15:52:14.142 INFO:tasks.workunit.client.0.smithi172.stderr:dumped pgs 2019-12-21T15:52:14.151 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: primary=2 2019-12-21T15:52:14.151 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:87: TEST_recover_unexpected: eval 'pg="3.e"' 2019-12-21T15:52:14.151 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:87: TEST_recover_unexpected: pg=3.e 2019-12-21T15:52:14.151 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: ceph tell 3.e scrub 2019-12-21T15:52:14.264 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:52:14.264 INFO:tasks.workunit.client.0.smithi172.stdout: "deep": false, 2019-12-21T15:52:14.265 INFO:tasks.workunit.client.0.smithi172.stdout: "stamp": "2019-12-14T15:50:34.263496+0000" 2019-12-21T15:52:14.265 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:52:14.273 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:84: TEST_recover_unexpected: for qpg in $(ceph pg dump pgs --format=json-pretty | jq '.pg_stats[].pgid') 2019-12-21T15:52:14.274 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: ceph pg dump pgs --format=json 2019-12-21T15:52:14.274 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: jq '.pg_stats[] | select(.pgid == "1.c") | .acting_primary' 2019-12-21T15:52:14.496 INFO:tasks.workunit.client.0.smithi172.stderr:dumped pgs 2019-12-21T15:52:14.505 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: primary=3 2019-12-21T15:52:14.505 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:87: TEST_recover_unexpected: eval 'pg="1.c"' 2019-12-21T15:52:14.505 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:87: TEST_recover_unexpected: pg=1.c 2019-12-21T15:52:14.506 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: ceph tell 1.c scrub 2019-12-21T15:52:14.649 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:52:14.649 INFO:tasks.workunit.client.0.smithi172.stdout: "deep": false, 2019-12-21T15:52:14.649 INFO:tasks.workunit.client.0.smithi172.stdout: "stamp": "2019-12-14T15:50:34.647688+0000" 2019-12-21T15:52:14.649 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:52:14.657 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:84: TEST_recover_unexpected: for qpg in $(ceph pg dump pgs --format=json-pretty | jq '.pg_stats[].pgid') 2019-12-21T15:52:14.658 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: ceph pg dump pgs --format=json 2019-12-21T15:52:14.658 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: jq '.pg_stats[] | select(.pgid == "2.f") | .acting_primary' 2019-12-21T15:52:14.882 INFO:tasks.workunit.client.0.smithi172.stderr:dumped pgs 2019-12-21T15:52:14.891 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: primary=0 2019-12-21T15:52:14.891 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:87: TEST_recover_unexpected: eval 'pg="2.f"' 2019-12-21T15:52:14.891 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:87: TEST_recover_unexpected: pg=2.f 2019-12-21T15:52:14.891 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: ceph tell 2.f scrub 2019-12-21T15:52:15.004 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:52:15.004 INFO:tasks.workunit.client.0.smithi172.stdout: "deep": false, 2019-12-21T15:52:15.005 INFO:tasks.workunit.client.0.smithi172.stdout: "stamp": "2019-12-14T15:50:35.003467+0000" 2019-12-21T15:52:15.005 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:52:15.013 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:84: TEST_recover_unexpected: for qpg in $(ceph pg dump pgs --format=json-pretty | jq '.pg_stats[].pgid') 2019-12-21T15:52:15.014 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: ceph pg dump pgs --format=json 2019-12-21T15:52:15.014 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: jq '.pg_stats[] | select(.pgid == "3.9") | .acting_primary' 2019-12-21T15:52:15.240 INFO:tasks.workunit.client.0.smithi172.stderr:dumped pgs 2019-12-21T15:52:15.249 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: primary=4 2019-12-21T15:52:15.249 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:87: TEST_recover_unexpected: eval 'pg="3.9"' 2019-12-21T15:52:15.249 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:87: TEST_recover_unexpected: pg=3.9 2019-12-21T15:52:15.250 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: ceph tell 3.9 scrub 2019-12-21T15:52:15.362 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:52:15.362 INFO:tasks.workunit.client.0.smithi172.stdout: "deep": false, 2019-12-21T15:52:15.362 INFO:tasks.workunit.client.0.smithi172.stdout: "stamp": "2019-12-14T15:50:35.361209+0000" 2019-12-21T15:52:15.363 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:52:15.370 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:84: TEST_recover_unexpected: for qpg in $(ceph pg dump pgs --format=json-pretty | jq '.pg_stats[].pgid') 2019-12-21T15:52:15.372 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: ceph pg dump pgs --format=json 2019-12-21T15:52:15.372 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: jq '.pg_stats[] | select(.pgid == "1.b") | .acting_primary' 2019-12-21T15:52:15.598 INFO:tasks.workunit.client.0.smithi172.stderr:dumped pgs 2019-12-21T15:52:15.606 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: primary=1 2019-12-21T15:52:15.606 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:87: TEST_recover_unexpected: eval 'pg="1.b"' 2019-12-21T15:52:15.607 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:87: TEST_recover_unexpected: pg=1.b 2019-12-21T15:52:15.607 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: ceph tell 1.b scrub 2019-12-21T15:52:15.720 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:52:15.720 INFO:tasks.workunit.client.0.smithi172.stdout: "deep": false, 2019-12-21T15:52:15.720 INFO:tasks.workunit.client.0.smithi172.stdout: "stamp": "2019-12-14T15:50:35.718811+0000" 2019-12-21T15:52:15.720 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:52:15.728 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:84: TEST_recover_unexpected: for qpg in $(ceph pg dump pgs --format=json-pretty | jq '.pg_stats[].pgid') 2019-12-21T15:52:15.729 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: ceph pg dump pgs --format=json 2019-12-21T15:52:15.729 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: jq '.pg_stats[] | select(.pgid == "2.8") | .acting_primary' 2019-12-21T15:52:15.956 INFO:tasks.workunit.client.0.smithi172.stderr:dumped pgs 2019-12-21T15:52:15.964 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: primary=0 2019-12-21T15:52:15.964 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:87: TEST_recover_unexpected: eval 'pg="2.8"' 2019-12-21T15:52:15.964 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:87: TEST_recover_unexpected: pg=2.8 2019-12-21T15:52:15.965 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: ceph tell 2.8 scrub 2019-12-21T15:52:16.079 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:52:16.079 INFO:tasks.workunit.client.0.smithi172.stdout: "deep": false, 2019-12-21T15:52:16.079 INFO:tasks.workunit.client.0.smithi172.stdout: "stamp": "2019-12-14T15:50:36.077652+0000" 2019-12-21T15:52:16.079 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:52:16.087 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:84: TEST_recover_unexpected: for qpg in $(ceph pg dump pgs --format=json-pretty | jq '.pg_stats[].pgid') 2019-12-21T15:52:16.088 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: ceph pg dump pgs --format=json 2019-12-21T15:52:16.088 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: jq '.pg_stats[] | select(.pgid == "3.8") | .acting_primary' 2019-12-21T15:52:16.311 INFO:tasks.workunit.client.0.smithi172.stderr:dumped pgs 2019-12-21T15:52:16.320 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: primary=4 2019-12-21T15:52:16.320 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:87: TEST_recover_unexpected: eval 'pg="3.8"' 2019-12-21T15:52:16.320 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:87: TEST_recover_unexpected: pg=3.8 2019-12-21T15:52:16.321 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: ceph tell 3.8 scrub 2019-12-21T15:52:16.434 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:52:16.434 INFO:tasks.workunit.client.0.smithi172.stdout: "deep": false, 2019-12-21T15:52:16.434 INFO:tasks.workunit.client.0.smithi172.stdout: "stamp": "2019-12-14T15:50:36.432117+0000" 2019-12-21T15:52:16.434 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:52:16.441 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:84: TEST_recover_unexpected: for qpg in $(ceph pg dump pgs --format=json-pretty | jq '.pg_stats[].pgid') 2019-12-21T15:52:16.442 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: ceph pg dump pgs --format=json 2019-12-21T15:52:16.442 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: jq '.pg_stats[] | select(.pgid == "1.a") | .acting_primary' 2019-12-21T15:52:16.667 INFO:tasks.workunit.client.0.smithi172.stderr:dumped pgs 2019-12-21T15:52:16.676 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: primary=1 2019-12-21T15:52:16.676 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:87: TEST_recover_unexpected: eval 'pg="1.a"' 2019-12-21T15:52:16.677 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:87: TEST_recover_unexpected: pg=1.a 2019-12-21T15:52:16.677 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: ceph tell 1.a scrub 2019-12-21T15:52:16.790 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:52:16.790 INFO:tasks.workunit.client.0.smithi172.stdout: "deep": false, 2019-12-21T15:52:16.790 INFO:tasks.workunit.client.0.smithi172.stdout: "stamp": "2019-12-14T15:50:36.788931+0000" 2019-12-21T15:52:16.790 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:52:16.798 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:84: TEST_recover_unexpected: for qpg in $(ceph pg dump pgs --format=json-pretty | jq '.pg_stats[].pgid') 2019-12-21T15:52:16.799 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: ceph pg dump pgs --format=json 2019-12-21T15:52:16.799 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: jq '.pg_stats[] | select(.pgid == "2.9") | .acting_primary' 2019-12-21T15:52:17.026 INFO:tasks.workunit.client.0.smithi172.stderr:dumped pgs 2019-12-21T15:52:17.035 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: primary=4 2019-12-21T15:52:17.035 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:87: TEST_recover_unexpected: eval 'pg="2.9"' 2019-12-21T15:52:17.035 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:87: TEST_recover_unexpected: pg=2.9 2019-12-21T15:52:17.035 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: ceph tell 2.9 scrub 2019-12-21T15:52:17.148 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:52:17.148 INFO:tasks.workunit.client.0.smithi172.stdout: "deep": false, 2019-12-21T15:52:17.149 INFO:tasks.workunit.client.0.smithi172.stdout: "stamp": "2019-12-14T15:50:37.147365+0000" 2019-12-21T15:52:17.149 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:52:17.156 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:84: TEST_recover_unexpected: for qpg in $(ceph pg dump pgs --format=json-pretty | jq '.pg_stats[].pgid') 2019-12-21T15:52:17.158 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: ceph pg dump pgs --format=json 2019-12-21T15:52:17.158 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: jq '.pg_stats[] | select(.pgid == "3.b") | .acting_primary' 2019-12-21T15:52:17.384 INFO:tasks.workunit.client.0.smithi172.stderr:dumped pgs 2019-12-21T15:52:17.391 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: primary=3 2019-12-21T15:52:17.391 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:87: TEST_recover_unexpected: eval 'pg="3.b"' 2019-12-21T15:52:17.391 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:87: TEST_recover_unexpected: pg=3.b 2019-12-21T15:52:17.391 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: ceph tell 3.b scrub 2019-12-21T15:52:17.504 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:52:17.505 INFO:tasks.workunit.client.0.smithi172.stdout: "deep": false, 2019-12-21T15:52:17.505 INFO:tasks.workunit.client.0.smithi172.stdout: "stamp": "2019-12-14T15:50:37.503448+0000" 2019-12-21T15:52:17.505 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:52:17.513 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:84: TEST_recover_unexpected: for qpg in $(ceph pg dump pgs --format=json-pretty | jq '.pg_stats[].pgid') 2019-12-21T15:52:17.515 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: ceph pg dump pgs --format=json 2019-12-21T15:52:17.515 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: jq '.pg_stats[] | select(.pgid == "1.9") | .acting_primary' 2019-12-21T15:52:17.738 INFO:tasks.workunit.client.0.smithi172.stderr:dumped pgs 2019-12-21T15:52:17.747 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: primary=1 2019-12-21T15:52:17.747 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:87: TEST_recover_unexpected: eval 'pg="1.9"' 2019-12-21T15:52:17.748 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:87: TEST_recover_unexpected: pg=1.9 2019-12-21T15:52:17.748 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: ceph tell 1.9 scrub 2019-12-21T15:52:17.861 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:52:17.861 INFO:tasks.workunit.client.0.smithi172.stdout: "deep": false, 2019-12-21T15:52:17.861 INFO:tasks.workunit.client.0.smithi172.stdout: "stamp": "2019-12-14T15:50:37.859855+0000" 2019-12-21T15:52:17.861 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:52:17.869 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:84: TEST_recover_unexpected: for qpg in $(ceph pg dump pgs --format=json-pretty | jq '.pg_stats[].pgid') 2019-12-21T15:52:17.870 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: ceph pg dump pgs --format=json 2019-12-21T15:52:17.870 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: jq '.pg_stats[] | select(.pgid == "2.a") | .acting_primary' 2019-12-21T15:52:18.097 INFO:tasks.workunit.client.0.smithi172.stderr:dumped pgs 2019-12-21T15:52:18.106 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: primary=1 2019-12-21T15:52:18.106 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:87: TEST_recover_unexpected: eval 'pg="2.a"' 2019-12-21T15:52:18.106 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:87: TEST_recover_unexpected: pg=2.a 2019-12-21T15:52:18.106 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: ceph tell 2.a scrub 2019-12-21T15:52:18.220 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:52:18.220 INFO:tasks.workunit.client.0.smithi172.stdout: "deep": false, 2019-12-21T15:52:18.220 INFO:tasks.workunit.client.0.smithi172.stdout: "stamp": "2019-12-14T15:50:38.219021+0000" 2019-12-21T15:52:18.220 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:52:18.228 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:84: TEST_recover_unexpected: for qpg in $(ceph pg dump pgs --format=json-pretty | jq '.pg_stats[].pgid') 2019-12-21T15:52:18.229 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: ceph pg dump pgs --format=json 2019-12-21T15:52:18.230 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: jq '.pg_stats[] | select(.pgid == "3.a") | .acting_primary' 2019-12-21T15:52:18.456 INFO:tasks.workunit.client.0.smithi172.stderr:dumped pgs 2019-12-21T15:52:18.464 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: primary=3 2019-12-21T15:52:18.465 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:87: TEST_recover_unexpected: eval 'pg="3.a"' 2019-12-21T15:52:18.465 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:87: TEST_recover_unexpected: pg=3.a 2019-12-21T15:52:18.465 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: ceph tell 3.a scrub 2019-12-21T15:52:18.577 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:52:18.577 INFO:tasks.workunit.client.0.smithi172.stdout: "deep": false, 2019-12-21T15:52:18.577 INFO:tasks.workunit.client.0.smithi172.stdout: "stamp": "2019-12-14T15:50:38.576046+0000" 2019-12-21T15:52:18.578 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:52:18.585 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:84: TEST_recover_unexpected: for qpg in $(ceph pg dump pgs --format=json-pretty | jq '.pg_stats[].pgid') 2019-12-21T15:52:18.586 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: ceph pg dump pgs --format=json 2019-12-21T15:52:18.587 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: jq '.pg_stats[] | select(.pgid == "1.8") | .acting_primary' 2019-12-21T15:52:18.810 INFO:tasks.workunit.client.0.smithi172.stderr:dumped pgs 2019-12-21T15:52:18.819 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: primary=5 2019-12-21T15:52:18.819 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:87: TEST_recover_unexpected: eval 'pg="1.8"' 2019-12-21T15:52:18.819 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:87: TEST_recover_unexpected: pg=1.8 2019-12-21T15:52:18.819 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: ceph tell 1.8 scrub 2019-12-21T15:52:18.932 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:52:18.932 INFO:tasks.workunit.client.0.smithi172.stdout: "deep": false, 2019-12-21T15:52:18.932 INFO:tasks.workunit.client.0.smithi172.stdout: "stamp": "2019-12-14T15:50:38.931060+0000" 2019-12-21T15:52:18.933 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:52:18.940 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:84: TEST_recover_unexpected: for qpg in $(ceph pg dump pgs --format=json-pretty | jq '.pg_stats[].pgid') 2019-12-21T15:52:18.941 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: ceph pg dump pgs --format=json 2019-12-21T15:52:18.941 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: jq '.pg_stats[] | select(.pgid == "2.b") | .acting_primary' 2019-12-21T15:52:19.165 INFO:tasks.workunit.client.0.smithi172.stderr:dumped pgs 2019-12-21T15:52:19.174 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: primary=3 2019-12-21T15:52:19.174 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:87: TEST_recover_unexpected: eval 'pg="2.b"' 2019-12-21T15:52:19.174 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:87: TEST_recover_unexpected: pg=2.b 2019-12-21T15:52:19.175 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: ceph tell 2.b scrub 2019-12-21T15:52:19.288 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:52:19.288 INFO:tasks.workunit.client.0.smithi172.stdout: "deep": false, 2019-12-21T15:52:19.288 INFO:tasks.workunit.client.0.smithi172.stdout: "stamp": "2019-12-14T15:50:39.286910+0000" 2019-12-21T15:52:19.288 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:52:19.296 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:84: TEST_recover_unexpected: for qpg in $(ceph pg dump pgs --format=json-pretty | jq '.pg_stats[].pgid') 2019-12-21T15:52:19.297 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: ceph pg dump pgs --format=json 2019-12-21T15:52:19.297 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: jq '.pg_stats[] | select(.pgid == "3.5") | .acting_primary' 2019-12-21T15:52:19.522 INFO:tasks.workunit.client.0.smithi172.stderr:dumped pgs 2019-12-21T15:52:19.531 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: primary=5 2019-12-21T15:52:19.531 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:87: TEST_recover_unexpected: eval 'pg="3.5"' 2019-12-21T15:52:19.531 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:87: TEST_recover_unexpected: pg=3.5 2019-12-21T15:52:19.531 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: ceph tell 3.5 scrub 2019-12-21T15:52:19.645 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:52:19.645 INFO:tasks.workunit.client.0.smithi172.stdout: "deep": false, 2019-12-21T15:52:19.645 INFO:tasks.workunit.client.0.smithi172.stdout: "stamp": "2019-12-14T15:50:39.644099+0000" 2019-12-21T15:52:19.645 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:52:19.653 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:84: TEST_recover_unexpected: for qpg in $(ceph pg dump pgs --format=json-pretty | jq '.pg_stats[].pgid') 2019-12-21T15:52:19.654 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: ceph pg dump pgs --format=json 2019-12-21T15:52:19.654 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: jq '.pg_stats[] | select(.pgid == "1.7") | .acting_primary' 2019-12-21T15:52:19.880 INFO:tasks.workunit.client.0.smithi172.stderr:dumped pgs 2019-12-21T15:52:19.889 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: primary=1 2019-12-21T15:52:19.890 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:87: TEST_recover_unexpected: eval 'pg="1.7"' 2019-12-21T15:52:19.890 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:87: TEST_recover_unexpected: pg=1.7 2019-12-21T15:52:19.890 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: ceph tell 1.7 scrub 2019-12-21T15:52:20.002 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:52:20.002 INFO:tasks.workunit.client.0.smithi172.stdout: "deep": false, 2019-12-21T15:52:20.002 INFO:tasks.workunit.client.0.smithi172.stdout: "stamp": "2019-12-14T15:50:40.000825+0000" 2019-12-21T15:52:20.003 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:52:20.011 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:84: TEST_recover_unexpected: for qpg in $(ceph pg dump pgs --format=json-pretty | jq '.pg_stats[].pgid') 2019-12-21T15:52:20.012 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: ceph pg dump pgs --format=json 2019-12-21T15:52:20.013 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: jq '.pg_stats[] | select(.pgid == "2.4") | .acting_primary' 2019-12-21T15:52:20.238 INFO:tasks.workunit.client.0.smithi172.stderr:dumped pgs 2019-12-21T15:52:20.248 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: primary=1 2019-12-21T15:52:20.248 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:87: TEST_recover_unexpected: eval 'pg="2.4"' 2019-12-21T15:52:20.248 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:87: TEST_recover_unexpected: pg=2.4 2019-12-21T15:52:20.248 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: ceph tell 2.4 scrub 2019-12-21T15:52:20.360 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:52:20.360 INFO:tasks.workunit.client.0.smithi172.stdout: "deep": false, 2019-12-21T15:52:20.360 INFO:tasks.workunit.client.0.smithi172.stdout: "stamp": "2019-12-14T15:50:40.359130+0000" 2019-12-21T15:52:20.361 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:52:20.368 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:84: TEST_recover_unexpected: for qpg in $(ceph pg dump pgs --format=json-pretty | jq '.pg_stats[].pgid') 2019-12-21T15:52:20.369 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: ceph pg dump pgs --format=json 2019-12-21T15:52:20.370 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: jq '.pg_stats[] | select(.pgid == "3.4") | .acting_primary' 2019-12-21T15:52:20.595 INFO:tasks.workunit.client.0.smithi172.stderr:dumped pgs 2019-12-21T15:52:20.604 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: primary=1 2019-12-21T15:52:20.604 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:87: TEST_recover_unexpected: eval 'pg="3.4"' 2019-12-21T15:52:20.604 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:87: TEST_recover_unexpected: pg=3.4 2019-12-21T15:52:20.604 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: ceph tell 3.4 scrub 2019-12-21T15:52:20.717 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:52:20.717 INFO:tasks.workunit.client.0.smithi172.stdout: "deep": false, 2019-12-21T15:52:20.717 INFO:tasks.workunit.client.0.smithi172.stdout: "stamp": "2019-12-14T15:50:40.716191+0000" 2019-12-21T15:52:20.717 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:52:20.725 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:84: TEST_recover_unexpected: for qpg in $(ceph pg dump pgs --format=json-pretty | jq '.pg_stats[].pgid') 2019-12-21T15:52:20.727 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: ceph pg dump pgs --format=json 2019-12-21T15:52:20.727 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: jq '.pg_stats[] | select(.pgid == "1.6") | .acting_primary' 2019-12-21T15:52:20.952 INFO:tasks.workunit.client.0.smithi172.stderr:dumped pgs 2019-12-21T15:52:20.961 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: primary=3 2019-12-21T15:52:20.962 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:87: TEST_recover_unexpected: eval 'pg="1.6"' 2019-12-21T15:52:20.962 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:87: TEST_recover_unexpected: pg=1.6 2019-12-21T15:52:20.962 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: ceph tell 1.6 scrub 2019-12-21T15:52:21.075 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:52:21.076 INFO:tasks.workunit.client.0.smithi172.stdout: "deep": false, 2019-12-21T15:52:21.076 INFO:tasks.workunit.client.0.smithi172.stdout: "stamp": "2019-12-14T15:50:41.074553+0000" 2019-12-21T15:52:21.076 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:52:21.084 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:84: TEST_recover_unexpected: for qpg in $(ceph pg dump pgs --format=json-pretty | jq '.pg_stats[].pgid') 2019-12-21T15:52:21.085 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: ceph pg dump pgs --format=json 2019-12-21T15:52:21.085 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: jq '.pg_stats[] | select(.pgid == "2.5") | .acting_primary' 2019-12-21T15:52:21.309 INFO:tasks.workunit.client.0.smithi172.stderr:dumped pgs 2019-12-21T15:52:21.318 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: primary=3 2019-12-21T15:52:21.318 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:87: TEST_recover_unexpected: eval 'pg="2.5"' 2019-12-21T15:52:21.318 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:87: TEST_recover_unexpected: pg=2.5 2019-12-21T15:52:21.318 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: ceph tell 2.5 scrub 2019-12-21T15:52:21.432 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:52:21.432 INFO:tasks.workunit.client.0.smithi172.stdout: "deep": false, 2019-12-21T15:52:21.432 INFO:tasks.workunit.client.0.smithi172.stdout: "stamp": "2019-12-14T15:50:41.430902+0000" 2019-12-21T15:52:21.432 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:52:21.440 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:84: TEST_recover_unexpected: for qpg in $(ceph pg dump pgs --format=json-pretty | jq '.pg_stats[].pgid') 2019-12-21T15:52:21.441 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: ceph pg dump pgs --format=json 2019-12-21T15:52:21.441 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: jq '.pg_stats[] | select(.pgid == "1.1") | .acting_primary' 2019-12-21T15:52:21.666 INFO:tasks.workunit.client.0.smithi172.stderr:dumped pgs 2019-12-21T15:52:21.675 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: primary=5 2019-12-21T15:52:21.675 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:87: TEST_recover_unexpected: eval 'pg="1.1"' 2019-12-21T15:52:21.675 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:87: TEST_recover_unexpected: pg=1.1 2019-12-21T15:52:21.675 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: ceph tell 1.1 scrub 2019-12-21T15:52:21.788 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:52:21.788 INFO:tasks.workunit.client.0.smithi172.stdout: "deep": false, 2019-12-21T15:52:21.788 INFO:tasks.workunit.client.0.smithi172.stdout: "stamp": "2019-12-14T15:50:41.787088+0000" 2019-12-21T15:52:21.788 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:52:21.796 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:84: TEST_recover_unexpected: for qpg in $(ceph pg dump pgs --format=json-pretty | jq '.pg_stats[].pgid') 2019-12-21T15:52:21.797 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: ceph pg dump pgs --format=json 2019-12-21T15:52:21.798 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: jq '.pg_stats[] | select(.pgid == "2.2") | .acting_primary' 2019-12-21T15:52:22.024 INFO:tasks.workunit.client.0.smithi172.stderr:dumped pgs 2019-12-21T15:52:22.033 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: primary=5 2019-12-21T15:52:22.033 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:87: TEST_recover_unexpected: eval 'pg="2.2"' 2019-12-21T15:52:22.033 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:87: TEST_recover_unexpected: pg=2.2 2019-12-21T15:52:22.033 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: ceph tell 2.2 scrub 2019-12-21T15:52:22.147 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:52:22.147 INFO:tasks.workunit.client.0.smithi172.stdout: "deep": false, 2019-12-21T15:52:22.147 INFO:tasks.workunit.client.0.smithi172.stdout: "stamp": "2019-12-14T15:50:42.145860+0000" 2019-12-21T15:52:22.147 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:52:22.155 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:84: TEST_recover_unexpected: for qpg in $(ceph pg dump pgs --format=json-pretty | jq '.pg_stats[].pgid') 2019-12-21T15:52:22.156 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: ceph pg dump pgs --format=json 2019-12-21T15:52:22.156 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: jq '.pg_stats[] | select(.pgid == "3.3") | .acting_primary' 2019-12-21T15:52:22.379 INFO:tasks.workunit.client.0.smithi172.stderr:dumped pgs 2019-12-21T15:52:22.388 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: primary=4 2019-12-21T15:52:22.388 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:87: TEST_recover_unexpected: eval 'pg="3.3"' 2019-12-21T15:52:22.388 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:87: TEST_recover_unexpected: pg=3.3 2019-12-21T15:52:22.389 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: ceph tell 3.3 scrub 2019-12-21T15:52:22.502 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:52:22.502 INFO:tasks.workunit.client.0.smithi172.stdout: "deep": false, 2019-12-21T15:52:22.502 INFO:tasks.workunit.client.0.smithi172.stdout: "stamp": "2019-12-14T15:50:42.500954+0000" 2019-12-21T15:52:22.502 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:52:22.510 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:84: TEST_recover_unexpected: for qpg in $(ceph pg dump pgs --format=json-pretty | jq '.pg_stats[].pgid') 2019-12-21T15:52:22.511 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: ceph pg dump pgs --format=json 2019-12-21T15:52:22.511 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: jq '.pg_stats[] | select(.pgid == "1.0") | .acting_primary' 2019-12-21T15:52:22.754 INFO:tasks.workunit.client.0.smithi172.stderr:dumped pgs 2019-12-21T15:52:22.763 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: primary=1 2019-12-21T15:52:22.763 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:87: TEST_recover_unexpected: eval 'pg="1.0"' 2019-12-21T15:52:22.764 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:87: TEST_recover_unexpected: pg=1.0 2019-12-21T15:52:22.764 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: ceph tell 1.0 scrub 2019-12-21T15:52:22.877 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:52:22.878 INFO:tasks.workunit.client.0.smithi172.stdout: "deep": false, 2019-12-21T15:52:22.878 INFO:tasks.workunit.client.0.smithi172.stdout: "stamp": "2019-12-14T15:50:42.876490+0000" 2019-12-21T15:52:22.878 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:52:22.886 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:84: TEST_recover_unexpected: for qpg in $(ceph pg dump pgs --format=json-pretty | jq '.pg_stats[].pgid') 2019-12-21T15:52:22.887 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: ceph pg dump pgs --format=json 2019-12-21T15:52:22.887 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: jq '.pg_stats[] | select(.pgid == "2.3") | .acting_primary' 2019-12-21T15:52:23.112 INFO:tasks.workunit.client.0.smithi172.stderr:dumped pgs 2019-12-21T15:52:23.121 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: primary=5 2019-12-21T15:52:23.121 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:87: TEST_recover_unexpected: eval 'pg="2.3"' 2019-12-21T15:52:23.121 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:87: TEST_recover_unexpected: pg=2.3 2019-12-21T15:52:23.122 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: ceph tell 2.3 scrub 2019-12-21T15:52:23.234 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:52:23.234 INFO:tasks.workunit.client.0.smithi172.stdout: "deep": false, 2019-12-21T15:52:23.234 INFO:tasks.workunit.client.0.smithi172.stdout: "stamp": "2019-12-14T15:50:43.233249+0000" 2019-12-21T15:52:23.235 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:52:23.242 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:84: TEST_recover_unexpected: for qpg in $(ceph pg dump pgs --format=json-pretty | jq '.pg_stats[].pgid') 2019-12-21T15:52:23.243 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: ceph pg dump pgs --format=json 2019-12-21T15:52:23.244 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: jq '.pg_stats[] | select(.pgid == "3.2") | .acting_primary' 2019-12-21T15:52:23.469 INFO:tasks.workunit.client.0.smithi172.stderr:dumped pgs 2019-12-21T15:52:23.477 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: primary=3 2019-12-21T15:52:23.478 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:87: TEST_recover_unexpected: eval 'pg="3.2"' 2019-12-21T15:52:23.478 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:87: TEST_recover_unexpected: pg=3.2 2019-12-21T15:52:23.478 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: ceph tell 3.2 scrub 2019-12-21T15:52:23.590 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:52:23.591 INFO:tasks.workunit.client.0.smithi172.stdout: "deep": false, 2019-12-21T15:52:23.591 INFO:tasks.workunit.client.0.smithi172.stdout: "stamp": "2019-12-14T15:50:43.589419+0000" 2019-12-21T15:52:23.591 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:52:23.598 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:84: TEST_recover_unexpected: for qpg in $(ceph pg dump pgs --format=json-pretty | jq '.pg_stats[].pgid') 2019-12-21T15:52:23.600 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: ceph pg dump pgs --format=json 2019-12-21T15:52:23.600 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: jq '.pg_stats[] | select(.pgid == "1.2") | .acting_primary' 2019-12-21T15:52:23.825 INFO:tasks.workunit.client.0.smithi172.stderr:dumped pgs 2019-12-21T15:52:23.833 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: primary=0 2019-12-21T15:52:23.834 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:87: TEST_recover_unexpected: eval 'pg="1.2"' 2019-12-21T15:52:23.834 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:87: TEST_recover_unexpected: pg=1.2 2019-12-21T15:52:23.834 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: ceph tell 1.2 scrub 2019-12-21T15:52:23.947 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:52:23.947 INFO:tasks.workunit.client.0.smithi172.stdout: "deep": false, 2019-12-21T15:52:23.947 INFO:tasks.workunit.client.0.smithi172.stdout: "stamp": "2019-12-14T15:50:43.946014+0000" 2019-12-21T15:52:23.947 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:52:23.955 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:84: TEST_recover_unexpected: for qpg in $(ceph pg dump pgs --format=json-pretty | jq '.pg_stats[].pgid') 2019-12-21T15:52:23.956 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: ceph pg dump pgs --format=json 2019-12-21T15:52:23.956 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: jq '.pg_stats[] | select(.pgid == "2.1") | .acting_primary' 2019-12-21T15:52:24.180 INFO:tasks.workunit.client.0.smithi172.stderr:dumped pgs 2019-12-21T15:52:24.189 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: primary=2 2019-12-21T15:52:24.189 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:87: TEST_recover_unexpected: eval 'pg="2.1"' 2019-12-21T15:52:24.189 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:87: TEST_recover_unexpected: pg=2.1 2019-12-21T15:52:24.189 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: ceph tell 2.1 scrub 2019-12-21T15:52:24.303 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:52:24.303 INFO:tasks.workunit.client.0.smithi172.stdout: "deep": false, 2019-12-21T15:52:24.304 INFO:tasks.workunit.client.0.smithi172.stdout: "stamp": "2019-12-14T15:50:44.302575+0000" 2019-12-21T15:52:24.304 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:52:24.312 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:84: TEST_recover_unexpected: for qpg in $(ceph pg dump pgs --format=json-pretty | jq '.pg_stats[].pgid') 2019-12-21T15:52:24.313 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: ceph pg dump pgs --format=json 2019-12-21T15:52:24.313 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: jq '.pg_stats[] | select(.pgid == "3.0") | .acting_primary' 2019-12-21T15:52:24.540 INFO:tasks.workunit.client.0.smithi172.stderr:dumped pgs 2019-12-21T15:52:24.547 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: primary=1 2019-12-21T15:52:24.547 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:87: TEST_recover_unexpected: eval 'pg="3.0"' 2019-12-21T15:52:24.547 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:87: TEST_recover_unexpected: pg=3.0 2019-12-21T15:52:24.547 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: ceph tell 3.0 scrub 2019-12-21T15:52:24.660 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:52:24.660 INFO:tasks.workunit.client.0.smithi172.stdout: "deep": false, 2019-12-21T15:52:24.660 INFO:tasks.workunit.client.0.smithi172.stdout: "stamp": "2019-12-14T15:50:44.658995+0000" 2019-12-21T15:52:24.660 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:52:24.668 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:84: TEST_recover_unexpected: for qpg in $(ceph pg dump pgs --format=json-pretty | jq '.pg_stats[].pgid') 2019-12-21T15:52:24.669 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: ceph pg dump pgs --format=json 2019-12-21T15:52:24.669 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: jq '.pg_stats[] | select(.pgid == "1.3") | .acting_primary' 2019-12-21T15:52:24.894 INFO:tasks.workunit.client.0.smithi172.stderr:dumped pgs 2019-12-21T15:52:24.903 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: primary=4 2019-12-21T15:52:24.903 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:87: TEST_recover_unexpected: eval 'pg="1.3"' 2019-12-21T15:52:24.903 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:87: TEST_recover_unexpected: pg=1.3 2019-12-21T15:52:24.903 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: ceph tell 1.3 scrub 2019-12-21T15:52:25.016 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:52:25.016 INFO:tasks.workunit.client.0.smithi172.stdout: "deep": false, 2019-12-21T15:52:25.016 INFO:tasks.workunit.client.0.smithi172.stdout: "stamp": "2019-12-14T15:50:45.014718+0000" 2019-12-21T15:52:25.016 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:52:25.024 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:84: TEST_recover_unexpected: for qpg in $(ceph pg dump pgs --format=json-pretty | jq '.pg_stats[].pgid') 2019-12-21T15:52:25.025 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: ceph pg dump pgs --format=json 2019-12-21T15:52:25.025 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: jq '.pg_stats[] | select(.pgid == "2.0") | .acting_primary' 2019-12-21T15:52:25.250 INFO:tasks.workunit.client.0.smithi172.stderr:dumped pgs 2019-12-21T15:52:25.259 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: primary=3 2019-12-21T15:52:25.259 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:87: TEST_recover_unexpected: eval 'pg="2.0"' 2019-12-21T15:52:25.259 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:87: TEST_recover_unexpected: pg=2.0 2019-12-21T15:52:25.259 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: ceph tell 2.0 scrub 2019-12-21T15:52:25.373 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:52:25.374 INFO:tasks.workunit.client.0.smithi172.stdout: "deep": false, 2019-12-21T15:52:25.374 INFO:tasks.workunit.client.0.smithi172.stdout: "stamp": "2019-12-14T15:50:45.372506+0000" 2019-12-21T15:52:25.374 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:52:25.382 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:84: TEST_recover_unexpected: for qpg in $(ceph pg dump pgs --format=json-pretty | jq '.pg_stats[].pgid') 2019-12-21T15:52:25.383 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: ceph pg dump pgs --format=json 2019-12-21T15:52:25.383 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: jq '.pg_stats[] | select(.pgid == "3.1") | .acting_primary' 2019-12-21T15:52:25.608 INFO:tasks.workunit.client.0.smithi172.stderr:dumped pgs 2019-12-21T15:52:25.617 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: primary=4 2019-12-21T15:52:25.617 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:87: TEST_recover_unexpected: eval 'pg="3.1"' 2019-12-21T15:52:25.617 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:87: TEST_recover_unexpected: pg=3.1 2019-12-21T15:52:25.617 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: ceph tell 3.1 scrub 2019-12-21T15:52:25.729 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:52:25.729 INFO:tasks.workunit.client.0.smithi172.stdout: "deep": false, 2019-12-21T15:52:25.729 INFO:tasks.workunit.client.0.smithi172.stdout: "stamp": "2019-12-14T15:50:45.728301+0000" 2019-12-21T15:52:25.729 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:52:25.737 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:84: TEST_recover_unexpected: for qpg in $(ceph pg dump pgs --format=json-pretty | jq '.pg_stats[].pgid') 2019-12-21T15:52:25.739 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: ceph pg dump pgs --format=json 2019-12-21T15:52:25.739 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: jq '.pg_stats[] | select(.pgid == "3.6") | .acting_primary' 2019-12-21T15:52:25.964 INFO:tasks.workunit.client.0.smithi172.stderr:dumped pgs 2019-12-21T15:52:25.972 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: primary=0 2019-12-21T15:52:25.973 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:87: TEST_recover_unexpected: eval 'pg="3.6"' 2019-12-21T15:52:25.973 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:87: TEST_recover_unexpected: pg=3.6 2019-12-21T15:52:25.973 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: ceph tell 3.6 scrub 2019-12-21T15:52:26.088 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:52:26.088 INFO:tasks.workunit.client.0.smithi172.stdout: "deep": false, 2019-12-21T15:52:26.088 INFO:tasks.workunit.client.0.smithi172.stdout: "stamp": "2019-12-14T15:50:46.086797+0000" 2019-12-21T15:52:26.088 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:52:26.096 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:84: TEST_recover_unexpected: for qpg in $(ceph pg dump pgs --format=json-pretty | jq '.pg_stats[].pgid') 2019-12-21T15:52:26.097 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: ceph pg dump pgs --format=json 2019-12-21T15:52:26.097 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: jq '.pg_stats[] | select(.pgid == "1.4") | .acting_primary' 2019-12-21T15:52:26.322 INFO:tasks.workunit.client.0.smithi172.stderr:dumped pgs 2019-12-21T15:52:26.331 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: primary=3 2019-12-21T15:52:26.331 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:87: TEST_recover_unexpected: eval 'pg="1.4"' 2019-12-21T15:52:26.331 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:87: TEST_recover_unexpected: pg=1.4 2019-12-21T15:52:26.331 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: ceph tell 1.4 scrub 2019-12-21T15:52:26.445 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:52:26.445 INFO:tasks.workunit.client.0.smithi172.stdout: "deep": false, 2019-12-21T15:52:26.445 INFO:tasks.workunit.client.0.smithi172.stdout: "stamp": "2019-12-14T15:50:46.443675+0000" 2019-12-21T15:52:26.445 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:52:26.453 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:84: TEST_recover_unexpected: for qpg in $(ceph pg dump pgs --format=json-pretty | jq '.pg_stats[].pgid') 2019-12-21T15:52:26.454 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: ceph pg dump pgs --format=json 2019-12-21T15:52:26.454 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: jq '.pg_stats[] | select(.pgid == "2.7") | .acting_primary' 2019-12-21T15:52:26.680 INFO:tasks.workunit.client.0.smithi172.stderr:dumped pgs 2019-12-21T15:52:26.689 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: primary=3 2019-12-21T15:52:26.689 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:87: TEST_recover_unexpected: eval 'pg="2.7"' 2019-12-21T15:52:26.689 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:87: TEST_recover_unexpected: pg=2.7 2019-12-21T15:52:26.689 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: ceph tell 2.7 scrub 2019-12-21T15:52:26.802 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:52:26.802 INFO:tasks.workunit.client.0.smithi172.stdout: "deep": false, 2019-12-21T15:52:26.802 INFO:tasks.workunit.client.0.smithi172.stdout: "stamp": "2019-12-14T15:50:46.801167+0000" 2019-12-21T15:52:26.802 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:52:26.810 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:84: TEST_recover_unexpected: for qpg in $(ceph pg dump pgs --format=json-pretty | jq '.pg_stats[].pgid') 2019-12-21T15:52:26.812 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: ceph pg dump pgs --format=json 2019-12-21T15:52:26.812 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: jq '.pg_stats[] | select(.pgid == "3.7") | .acting_primary' 2019-12-21T15:52:27.036 INFO:tasks.workunit.client.0.smithi172.stderr:dumped pgs 2019-12-21T15:52:27.045 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: primary=4 2019-12-21T15:52:27.045 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:87: TEST_recover_unexpected: eval 'pg="3.7"' 2019-12-21T15:52:27.045 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:87: TEST_recover_unexpected: pg=3.7 2019-12-21T15:52:27.045 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: ceph tell 3.7 scrub 2019-12-21T15:52:27.157 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:52:27.158 INFO:tasks.workunit.client.0.smithi172.stdout: "deep": false, 2019-12-21T15:52:27.158 INFO:tasks.workunit.client.0.smithi172.stdout: "stamp": "2019-12-14T15:50:47.156376+0000" 2019-12-21T15:52:27.158 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:52:27.166 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:84: TEST_recover_unexpected: for qpg in $(ceph pg dump pgs --format=json-pretty | jq '.pg_stats[].pgid') 2019-12-21T15:52:27.167 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: ceph pg dump pgs --format=json 2019-12-21T15:52:27.167 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: jq '.pg_stats[] | select(.pgid == "1.5") | .acting_primary' 2019-12-21T15:52:27.391 INFO:tasks.workunit.client.0.smithi172.stderr:dumped pgs 2019-12-21T15:52:27.399 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: primary=4 2019-12-21T15:52:27.399 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:87: TEST_recover_unexpected: eval 'pg="1.5"' 2019-12-21T15:52:27.399 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:87: TEST_recover_unexpected: pg=1.5 2019-12-21T15:52:27.399 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: ceph tell 1.5 scrub 2019-12-21T15:52:27.512 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:52:27.512 INFO:tasks.workunit.client.0.smithi172.stdout: "deep": false, 2019-12-21T15:52:27.512 INFO:tasks.workunit.client.0.smithi172.stdout: "stamp": "2019-12-14T15:50:47.510576+0000" 2019-12-21T15:52:27.512 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:52:27.520 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:84: TEST_recover_unexpected: for qpg in $(ceph pg dump pgs --format=json-pretty | jq '.pg_stats[].pgid') 2019-12-21T15:52:27.521 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: ceph pg dump pgs --format=json 2019-12-21T15:52:27.521 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: jq '.pg_stats[] | select(.pgid == "2.6") | .acting_primary' 2019-12-21T15:52:27.748 INFO:tasks.workunit.client.0.smithi172.stderr:dumped pgs 2019-12-21T15:52:27.757 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:86: TEST_recover_unexpected: primary=1 2019-12-21T15:52:27.757 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:87: TEST_recover_unexpected: eval 'pg="2.6"' 2019-12-21T15:52:27.758 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:87: TEST_recover_unexpected: pg=2.6 2019-12-21T15:52:27.758 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:88: TEST_recover_unexpected: ceph tell 2.6 scrub 2019-12-21T15:52:27.870 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:52:27.870 INFO:tasks.workunit.client.0.smithi172.stdout: "deep": false, 2019-12-21T15:52:27.870 INFO:tasks.workunit.client.0.smithi172.stdout: "stamp": "2019-12-14T15:50:47.869116+0000" 2019-12-21T15:52:27.871 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:52:27.878 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:91: TEST_recover_unexpected: ceph pg dump pgs 2019-12-21T15:52:28.099 INFO:tasks.workunit.client.0.smithi172.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2019-12-21T15:52:28.099 INFO:tasks.workunit.client.0.smithi172.stdout:3.d 67 0 0 0 0 548864 0 0 67 67 active+clean 2019-12-21T15:48:53.596654+0000 46'67 48:82 [3,1,0] 3 [3,1,0] 3 0'0 2019-12-14T15:50:31.047070+0000 0'0 2019-12-21T15:48:52.433608+0000 0 2019-12-21T15:52:28.099 INFO:tasks.workunit.client.0.smithi172.stdout:1.f 62 0 0 0 0 507904 0 0 62 62 active+clean 2019-12-21T15:48:50.267215+0000 46'62 48:84 [3,0,4] 3 [3,0,4] 3 0'0 2019-12-14T15:50:31.406387+0000 0'0 2019-12-21T15:48:49.587218+0000 0 2019-12-21T15:52:28.099 INFO:tasks.workunit.client.0.smithi172.stdout:2.c 60 0 0 0 0 491520 0 0 60 60 active+clean 2019-12-21T15:48:51.921595+0000 46'60 48:77 [2,0,5] 2 [2,0,5] 2 0'0 2019-12-14T15:50:31.765715+0000 0'0 2019-12-21T15:48:51.001996+0000 0 2019-12-21T15:52:28.100 INFO:tasks.workunit.client.0.smithi172.stdout:3.c 60 0 0 0 0 491520 0 0 60 60 active+clean 2019-12-21T15:48:53.573137+0000 46'60 48:73 [5,3,1] 5 [5,3,1] 5 0'0 2019-12-14T15:50:32.121892+0000 0'0 2019-12-21T15:48:52.433608+0000 0 2019-12-21T15:52:28.100 INFO:tasks.workunit.client.0.smithi172.stdout:1.e 69 0 0 0 0 565248 0 0 69 69 active+clean 2019-12-21T15:48:50.233389+0000 46'69 48:92 [1,4,2] 1 [1,4,2] 1 0'0 2019-12-14T15:50:32.477832+0000 0'0 2019-12-21T15:48:49.587218+0000 0 2019-12-21T15:52:28.100 INFO:tasks.workunit.client.0.smithi172.stdout:2.d 67 0 0 0 0 548864 0 0 67 67 active+clean 2019-12-21T15:48:51.831821+0000 46'67 48:83 [5,1,3] 5 [5,1,3] 5 0'0 2019-12-14T15:50:32.835272+0000 0'0 2019-12-21T15:48:51.001996+0000 0 2019-12-21T15:52:28.100 INFO:tasks.workunit.client.0.smithi172.stdout:3.f 62 0 0 0 0 507904 0 0 62 62 active+clean 2019-12-21T15:48:53.597626+0000 46'62 48:76 [0,1,4] 0 [0,1,4] 0 0'0 2019-12-14T15:50:33.192542+0000 0'0 2019-12-21T15:48:52.433608+0000 0 2019-12-21T15:52:28.100 INFO:tasks.workunit.client.0.smithi172.stdout:1.d 67 0 0 0 0 548864 0 0 67 67 active+clean 2019-12-21T15:48:50.232330+0000 46'67 48:87 [2,4,1] 2 [2,4,1] 2 0'0 2019-12-14T15:50:33.550430+0000 0'0 2019-12-21T15:48:49.587218+0000 0 2019-12-21T15:52:28.100 INFO:tasks.workunit.client.0.smithi172.stdout:2.e 69 0 0 0 0 565248 0 0 69 69 active+clean 2019-12-21T15:48:51.921425+0000 46'69 48:86 [2,3,5] 2 [2,3,5] 2 0'0 2019-12-14T15:50:33.906226+0000 0'0 2019-12-21T15:48:51.001996+0000 0 2019-12-21T15:52:28.100 INFO:tasks.workunit.client.0.smithi172.stdout:3.e 69 0 0 0 0 565248 0 0 69 69 active+clean 2019-12-21T15:48:53.597334+0000 46'69 48:82 [2,4,1] 2 [2,4,1] 2 0'0 2019-12-14T15:50:34.263496+0000 0'0 2019-12-21T15:48:52.433608+0000 0 2019-12-21T15:52:28.101 INFO:tasks.workunit.client.0.smithi172.stdout:1.c 60 0 0 0 0 491520 0 0 60 60 active+clean 2019-12-21T15:48:50.266642+0000 46'60 48:82 [3,0,4] 3 [3,0,4] 3 0'0 2019-12-14T15:50:34.647688+0000 0'0 2019-12-21T15:48:49.587218+0000 0 2019-12-21T15:52:28.101 INFO:tasks.workunit.client.0.smithi172.stdout:2.f 62 0 0 0 0 507904 0 0 62 62 active+clean 2019-12-21T15:48:51.780390+0000 46'62 48:80 [0,5,4] 0 [0,5,4] 0 0'0 2019-12-14T15:50:35.003467+0000 0'0 2019-12-21T15:48:51.001996+0000 0 2019-12-21T15:52:28.101 INFO:tasks.workunit.client.0.smithi172.stdout:3.9 61 0 0 0 0 499712 0 0 61 61 active+clean 2019-12-21T15:48:53.597754+0000 46'61 48:76 [4,2,5] 4 [4,2,5] 4 0'0 2019-12-14T15:50:35.361209+0000 0'0 2019-12-21T15:48:52.433608+0000 0 2019-12-21T15:52:28.101 INFO:tasks.workunit.client.0.smithi172.stdout:1.b 65 0 0 0 0 532480 0 0 65 65 active+clean 2019-12-21T15:48:50.266884+0000 46'65 48:88 [1,0,2] 1 [1,0,2] 1 0'0 2019-12-14T15:50:35.718811+0000 0'0 2019-12-21T15:48:49.587218+0000 0 2019-12-21T15:52:28.101 INFO:tasks.workunit.client.0.smithi172.stdout:2.8 66 0 0 0 0 540672 0 0 66 66 active+clean 2019-12-21T15:48:51.779146+0000 46'66 48:84 [0,2,1] 0 [0,2,1] 0 0'0 2019-12-14T15:50:36.077652+0000 0'0 2019-12-21T15:48:51.001996+0000 0 2019-12-21T15:52:28.101 INFO:tasks.workunit.client.0.smithi172.stdout:3.8 66 0 0 0 0 540672 0 0 66 66 active+clean 2019-12-21T15:48:53.596865+0000 46'66 48:81 [4,1,2] 4 [4,1,2] 4 0'0 2019-12-14T15:50:36.432117+0000 0'0 2019-12-21T15:48:52.433608+0000 0 2019-12-21T15:52:28.101 INFO:tasks.workunit.client.0.smithi172.stdout:1.a 61 0 0 0 0 499712 0 0 61 61 active+clean 2019-12-21T15:48:50.232998+0000 46'61 48:84 [1,2,4] 1 [1,2,4] 1 0'0 2019-12-14T15:50:36.788931+0000 0'0 2019-12-21T15:48:49.587218+0000 0 2019-12-21T15:52:28.102 INFO:tasks.workunit.client.0.smithi172.stdout:2.9 61 0 0 0 0 499712 0 0 61 61 active+clean 2019-12-21T15:48:51.883247+0000 46'61 48:79 [4,1,0] 4 [4,1,0] 4 0'0 2019-12-14T15:50:37.147365+0000 0'0 2019-12-21T15:48:51.001996+0000 0 2019-12-21T15:52:28.102 INFO:tasks.workunit.client.0.smithi172.stdout:3.b 65 0 0 0 0 532480 0 0 65 65 active+clean 2019-12-21T15:48:53.598249+0000 46'65 48:80 [3,0,4] 3 [3,0,4] 3 0'0 2019-12-14T15:50:37.503448+0000 0'0 2019-12-21T15:48:52.433608+0000 0 2019-12-21T15:52:28.102 INFO:tasks.workunit.client.0.smithi172.stdout:1.9 61 0 0 0 0 499712 0 0 61 61 active+clean 2019-12-21T15:48:50.190683+0000 46'61 48:84 [1,4,5] 1 [1,4,5] 1 0'0 2019-12-14T15:50:37.859855+0000 0'0 2019-12-21T15:48:49.587218+0000 0 2019-12-21T15:52:28.102 INFO:tasks.workunit.client.0.smithi172.stdout:2.a 61 0 0 0 0 499712 0 0 61 61 active+clean 2019-12-21T15:48:51.921320+0000 46'61 48:80 [1,5,0] 1 [1,5,0] 1 0'0 2019-12-14T15:50:38.219021+0000 0'0 2019-12-21T15:48:51.001996+0000 0 2019-12-21T15:52:28.102 INFO:tasks.workunit.client.0.smithi172.stdout:3.a 61 0 0 0 0 499712 0 0 61 61 active+clean 2019-12-21T15:48:53.573078+0000 46'61 48:76 [3,1,5] 3 [3,1,5] 3 0'0 2019-12-14T15:50:38.576046+0000 0'0 2019-12-21T15:48:52.433608+0000 0 2019-12-21T15:52:28.102 INFO:tasks.workunit.client.0.smithi172.stdout:1.8 66 0 0 0 0 540672 0 0 66 66 active+clean 2019-12-21T15:48:51.230728+0000 46'66 48:86 [5,4,3] 5 [5,4,3] 5 0'0 2019-12-14T15:50:38.931060+0000 0'0 2019-12-21T15:48:49.587218+0000 0 2019-12-21T15:52:28.103 INFO:tasks.workunit.client.0.smithi172.stdout:2.b 65 0 0 0 0 532480 0 0 65 65 active+clean 2019-12-21T15:48:51.922814+0000 46'65 48:85 [3,2,5] 3 [3,2,5] 3 0'0 2019-12-14T15:50:39.286910+0000 0'0 2019-12-21T15:48:51.001996+0000 0 2019-12-21T15:52:28.103 INFO:tasks.workunit.client.0.smithi172.stdout:3.5 73 0 0 0 0 598016 0 0 73 73 active+clean 2019-12-21T15:48:53.483945+0000 46'73 48:86 [5,3,4] 5 [5,3,4] 5 0'0 2019-12-14T15:50:39.644099+0000 0'0 2019-12-21T15:48:52.433608+0000 0 2019-12-21T15:52:28.103 INFO:tasks.workunit.client.0.smithi172.stdout:1.7 53 0 0 0 0 434176 0 0 53 53 active+clean 2019-12-21T15:48:50.267398+0000 46'53 48:76 [1,3,0] 1 [1,3,0] 1 0'0 2019-12-14T15:50:40.000825+0000 0'0 2019-12-21T15:48:49.587218+0000 0 2019-12-21T15:52:28.103 INFO:tasks.workunit.client.0.smithi172.stdout:2.4 51 0 0 0 0 417792 0 0 51 51 active+clean 2019-12-21T15:48:51.883549+0000 46'51 48:70 [1,0,2] 1 [1,0,2] 1 0'0 2019-12-14T15:50:40.359130+0000 0'0 2019-12-21T15:48:51.001996+0000 0 2019-12-21T15:52:28.104 INFO:tasks.workunit.client.0.smithi172.stdout:3.4 51 0 0 0 0 417792 0 0 51 51 active+clean 2019-12-21T15:48:53.596868+0000 46'51 48:66 [1,2,4] 1 [1,2,4] 1 0'0 2019-12-14T15:50:40.716191+0000 0'0 2019-12-21T15:48:52.433608+0000 0 2019-12-21T15:52:28.104 INFO:tasks.workunit.client.0.smithi172.stdout:1.6 64 0 0 0 0 524288 0 0 64 64 active+clean 2019-12-21T15:48:50.232532+0000 46'64 48:86 [3,4,2] 3 [3,4,2] 3 0'0 2019-12-14T15:50:41.074553+0000 0'0 2019-12-21T15:48:49.587218+0000 0 2019-12-21T15:52:28.104 INFO:tasks.workunit.client.0.smithi172.stdout:2.5 73 0 0 0 0 598016 0 0 73 73 active+clean 2019-12-21T15:48:51.922004+0000 46'73 48:93 [3,4,0] 3 [3,4,0] 3 0'0 2019-12-14T15:50:41.430902+0000 0'0 2019-12-21T15:48:51.001996+0000 0 2019-12-21T15:52:28.104 INFO:tasks.workunit.client.0.smithi172.stdout:1.1 67 0 0 0 0 548864 0 0 67 67 active+clean 2019-12-21T15:48:50.513172+0000 46'67 48:87 [5,0,4] 5 [5,0,4] 5 0'0 2019-12-14T15:50:41.787088+0000 0'0 2019-12-21T15:48:49.587218+0000 0 2019-12-21T15:52:28.104 INFO:tasks.workunit.client.0.smithi172.stdout:2.2 55 0 0 0 0 450560 0 0 55 55 active+clean 2019-12-21T15:48:51.882192+0000 46'55 48:71 [5,1,0] 5 [5,1,0] 5 0'0 2019-12-14T15:50:42.145860+0000 0'0 2019-12-21T15:48:51.001996+0000 0 2019-12-21T15:52:28.104 INFO:tasks.workunit.client.0.smithi172.stdout:3.3 54 0 0 0 0 442368 0 0 54 54 active+clean 2019-12-21T15:48:53.573324+0000 46'54 47:68 [4,0,5] 4 [4,0,5] 4 0'0 2019-12-21T15:48:52.433608+0000 0'0 2019-12-21T15:48:52.433608+0000 0 2019-12-21T15:52:28.105 INFO:tasks.workunit.client.0.smithi172.stdout:1.0 72 0 0 0 0 589824 0 0 72 72 active+clean 2019-12-21T15:48:50.266436+0000 46'72 47:94 [1,0,2] 1 [1,0,2] 1 0'0 2019-12-21T15:48:49.587218+0000 0'0 2019-12-21T15:48:49.587218+0000 0 2019-12-21T15:52:28.105 INFO:tasks.workunit.client.0.smithi172.stdout:2.3 54 0 0 0 0 442368 0 0 54 54 active+clean 2019-12-21T15:48:51.921328+0000 46'54 48:70 [5,4,1] 5 [5,4,1] 5 0'0 2019-12-14T15:50:43.233249+0000 0'0 2019-12-21T15:48:51.001996+0000 0 2019-12-21T15:52:28.105 INFO:tasks.workunit.client.0.smithi172.stdout:3.2 55 0 0 0 0 450560 0 0 55 55 active+clean 2019-12-21T15:48:53.597372+0000 46'55 48:70 [3,5,0] 3 [3,5,0] 3 0'0 2019-12-14T15:50:43.589419+0000 0'0 2019-12-21T15:48:52.433608+0000 0 2019-12-21T15:52:28.105 INFO:tasks.workunit.client.0.smithi172.stdout:1.2 55 0 0 0 0 450560 0 0 55 55 active+clean 2019-12-21T15:48:50.266115+0000 46'55 48:77 [0,4,1] 0 [0,4,1] 0 0'0 2019-12-14T15:50:43.946014+0000 0'0 2019-12-21T15:48:49.587218+0000 0 2019-12-21T15:52:28.105 INFO:tasks.workunit.client.0.smithi172.stdout:2.1 67 0 0 0 0 548864 0 0 67 67 active+clean 2019-12-21T15:48:51.921897+0000 46'67 46:83 [2,5,0] 2 [2,5,0] 2 0'0 2019-12-21T15:48:51.001996+0000 0'0 2019-12-21T15:48:51.001996+0000 0 2019-12-21T15:52:28.105 INFO:tasks.workunit.client.0.smithi172.stdout:3.0 72 0 0 0 0 589824 0 0 72 72 active+clean 2019-12-21T15:48:53.598105+0000 46'72 47:86 [1,2,4] 1 [1,2,4] 1 0'0 2019-12-21T15:48:52.433608+0000 0'0 2019-12-21T15:48:52.433608+0000 0 2019-12-21T15:52:28.105 INFO:tasks.workunit.client.0.smithi172.stdout:1.3 54 0 0 0 0 442368 0 0 54 54 active+clean 2019-12-21T15:48:50.232614+0000 46'54 47:74 [4,1,2] 4 [4,1,2] 4 0'0 2019-12-21T15:48:49.587218+0000 0'0 2019-12-21T15:48:49.587218+0000 0 2019-12-21T15:52:28.106 INFO:tasks.workunit.client.0.smithi172.stdout:2.0 72 0 0 0 0 589824 0 0 72 72 active+clean 2019-12-21T15:48:51.921796+0000 46'72 48:92 [3,5,1] 3 [3,5,1] 3 0'0 2019-12-14T15:50:45.372506+0000 0'0 2019-12-21T15:48:51.001996+0000 0 2019-12-21T15:52:28.106 INFO:tasks.workunit.client.0.smithi172.stdout:3.1 67 0 0 0 0 548864 0 0 67 67 active+clean 2019-12-21T15:48:53.483350+0000 46'67 47:81 [4,3,5] 4 [4,3,5] 4 0'0 2019-12-21T15:48:52.433608+0000 0'0 2019-12-21T15:48:52.433608+0000 0 2019-12-21T15:52:28.106 INFO:tasks.workunit.client.0.smithi172.stdout:3.6 64 0 0 0 0 524288 0 0 64 64 active+clean 2019-12-21T15:48:53.596770+0000 46'64 47:77 [0,1,4] 0 [0,1,4] 0 0'0 2019-12-21T15:48:52.433608+0000 0'0 2019-12-21T15:48:52.433608+0000 0 2019-12-21T15:52:28.106 INFO:tasks.workunit.client.0.smithi172.stdout:1.4 51 0 0 0 0 417792 0 0 51 51 active+clean 2019-12-21T15:48:50.267005+0000 46'51 47:72 [3,0,1] 3 [3,0,1] 3 0'0 2019-12-21T15:48:49.587218+0000 0'0 2019-12-21T15:48:49.587218+0000 0 2019-12-21T15:52:28.106 INFO:tasks.workunit.client.0.smithi172.stdout:2.7 53 0 0 0 0 434176 0 0 53 53 active+clean 2019-12-21T15:48:51.882809+0000 46'53 47:72 [3,0,2] 3 [3,0,2] 3 0'0 2019-12-21T15:48:51.001996+0000 0'0 2019-12-21T15:48:51.001996+0000 0 2019-12-21T15:52:28.106 INFO:tasks.workunit.client.0.smithi172.stdout:3.7 53 0 0 0 0 434176 0 0 53 53 active+clean 2019-12-21T15:48:53.573718+0000 46'53 47:67 [4,3,0] 4 [4,3,0] 4 0'0 2019-12-21T15:48:52.433608+0000 0'0 2019-12-21T15:48:52.433608+0000 0 2019-12-21T15:52:28.106 INFO:tasks.workunit.client.0.smithi172.stdout:1.5 73 0 0 0 0 598016 0 0 73 73 active+clean 2019-12-21T15:48:50.232808+0000 46'73 47:93 [4,2,5] 4 [4,2,5] 4 0'0 2019-12-21T15:48:49.587218+0000 0'0 2019-12-21T15:48:49.587218+0000 0 2019-12-21T15:52:28.107 INFO:tasks.workunit.client.0.smithi172.stdout:2.6 64 0 0 0 0 524288 0 0 64 64 active+clean 2019-12-21T15:48:51.921603+0000 46'64 47:82 [1,0,4] 1 [1,0,4] 1 0'0 2019-12-21T15:48:51.001996+0000 0'0 2019-12-21T15:48:51.001996+0000 0 2019-12-21T15:52:28.107 INFO:tasks.workunit.client.0.smithi172.stdout: 2019-12-21T15:52:28.107 INFO:tasks.workunit.client.0.smithi172.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilisation. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2019-12-21T15:52:28.107 INFO:tasks.workunit.client.0.smithi172.stderr:dumped pgs 2019-12-21T15:52:28.110 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:93: TEST_recover_unexpected: jq .osd_max_scrubs 2019-12-21T15:52:28.110 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:93: TEST_recover_unexpected: get_asok_path osd.0 2019-12-21T15:52:28.111 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2019-12-21T15:52:28.111 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2019-12-21T15:52:28.111 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:52:28.111 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:52:28.111 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:52:28.112 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.0.asok 2019-12-21T15:52:28.112 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:93: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:52:28.112 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:93: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.0.asok dump_scrub_reservations 2019-12-21T15:52:28.205 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:93: TEST_recover_unexpected: max=4 2019-12-21T15:52:28.206 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:94: TEST_recover_unexpected: '[' 4 '!=' '4]' 2019-12-21T15:52:28.206 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh: line 94: [: missing `]' 2019-12-21T15:52:28.206 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:100: TEST_recover_unexpected: ceph osd unset noscrub 2019-12-21T15:52:28.587 INFO:tasks.workunit.client.0.smithi172.stderr:noscrub is unset 2019-12-21T15:52:28.595 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:102: TEST_recover_unexpected: ok=false 2019-12-21T15:52:28.596 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:103: TEST_recover_unexpected: seq 0 300 2019-12-21T15:52:28.597 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:103: TEST_recover_unexpected: for i in $(seq 0 300) 2019-12-21T15:52:28.597 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:105: TEST_recover_unexpected: ceph pg dump pgs 2019-12-21T15:52:28.820 INFO:tasks.workunit.client.0.smithi172.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2019-12-21T15:52:28.820 INFO:tasks.workunit.client.0.smithi172.stdout:3.d 67 0 0 0 0 548864 0 0 67 67 active+clean 2019-12-21T15:48:53.596654+0000 46'67 48:82 [3,1,0] 3 [3,1,0] 3 0'0 2019-12-14T15:50:31.047070+0000 0'0 2019-12-21T15:48:52.433608+0000 0 2019-12-21T15:52:28.820 INFO:tasks.workunit.client.0.smithi172.stdout:1.f 62 0 0 0 0 507904 0 0 62 62 active+clean 2019-12-21T15:48:50.267215+0000 46'62 48:84 [3,0,4] 3 [3,0,4] 3 0'0 2019-12-14T15:50:31.406387+0000 0'0 2019-12-21T15:48:49.587218+0000 0 2019-12-21T15:52:28.820 INFO:tasks.workunit.client.0.smithi172.stdout:2.c 60 0 0 0 0 491520 0 0 60 60 active+clean 2019-12-21T15:48:51.921595+0000 46'60 48:77 [2,0,5] 2 [2,0,5] 2 0'0 2019-12-14T15:50:31.765715+0000 0'0 2019-12-21T15:48:51.001996+0000 0 2019-12-21T15:52:28.820 INFO:tasks.workunit.client.0.smithi172.stdout:3.c 60 0 0 0 0 491520 0 0 60 60 active+clean 2019-12-21T15:48:53.573137+0000 46'60 48:73 [5,3,1] 5 [5,3,1] 5 0'0 2019-12-14T15:50:32.121892+0000 0'0 2019-12-21T15:48:52.433608+0000 0 2019-12-21T15:52:28.821 INFO:tasks.workunit.client.0.smithi172.stdout:1.e 69 0 0 0 0 565248 0 0 69 69 active+clean 2019-12-21T15:48:50.233389+0000 46'69 48:92 [1,4,2] 1 [1,4,2] 1 0'0 2019-12-14T15:50:32.477832+0000 0'0 2019-12-21T15:48:49.587218+0000 0 2019-12-21T15:52:28.821 INFO:tasks.workunit.client.0.smithi172.stdout:2.d 67 0 0 0 0 548864 0 0 67 67 active+clean 2019-12-21T15:48:51.831821+0000 46'67 48:83 [5,1,3] 5 [5,1,3] 5 0'0 2019-12-14T15:50:32.835272+0000 0'0 2019-12-21T15:48:51.001996+0000 0 2019-12-21T15:52:28.821 INFO:tasks.workunit.client.0.smithi172.stdout:3.f 62 0 0 0 0 507904 0 0 62 62 active+clean 2019-12-21T15:48:53.597626+0000 46'62 48:76 [0,1,4] 0 [0,1,4] 0 0'0 2019-12-14T15:50:33.192542+0000 0'0 2019-12-21T15:48:52.433608+0000 0 2019-12-21T15:52:28.821 INFO:tasks.workunit.client.0.smithi172.stdout:1.d 67 0 0 0 0 548864 0 0 67 67 active+clean 2019-12-21T15:48:50.232330+0000 46'67 48:87 [2,4,1] 2 [2,4,1] 2 0'0 2019-12-14T15:50:33.550430+0000 0'0 2019-12-21T15:48:49.587218+0000 0 2019-12-21T15:52:28.821 INFO:tasks.workunit.client.0.smithi172.stdout:2.e 69 0 0 0 0 565248 0 0 69 69 active+clean 2019-12-21T15:48:51.921425+0000 46'69 48:86 [2,3,5] 2 [2,3,5] 2 0'0 2019-12-14T15:50:33.906226+0000 0'0 2019-12-21T15:48:51.001996+0000 0 2019-12-21T15:52:28.821 INFO:tasks.workunit.client.0.smithi172.stdout:3.e 69 0 0 0 0 565248 0 0 69 69 active+clean 2019-12-21T15:48:53.597334+0000 46'69 48:82 [2,4,1] 2 [2,4,1] 2 0'0 2019-12-14T15:50:34.263496+0000 0'0 2019-12-21T15:48:52.433608+0000 0 2019-12-21T15:52:28.821 INFO:tasks.workunit.client.0.smithi172.stdout:1.c 60 0 0 0 0 491520 0 0 60 60 active+clean 2019-12-21T15:48:50.266642+0000 46'60 48:82 [3,0,4] 3 [3,0,4] 3 0'0 2019-12-14T15:50:34.647688+0000 0'0 2019-12-21T15:48:49.587218+0000 0 2019-12-21T15:52:28.821 INFO:tasks.workunit.client.0.smithi172.stdout:2.f 62 0 0 0 0 507904 0 0 62 62 active+clean 2019-12-21T15:48:51.780390+0000 46'62 48:80 [0,5,4] 0 [0,5,4] 0 0'0 2019-12-14T15:50:35.003467+0000 0'0 2019-12-21T15:48:51.001996+0000 0 2019-12-21T15:52:28.822 INFO:tasks.workunit.client.0.smithi172.stdout:3.9 61 0 0 0 0 499712 0 0 61 61 active+clean 2019-12-21T15:48:53.597754+0000 46'61 48:76 [4,2,5] 4 [4,2,5] 4 0'0 2019-12-14T15:50:35.361209+0000 0'0 2019-12-21T15:48:52.433608+0000 0 2019-12-21T15:52:28.822 INFO:tasks.workunit.client.0.smithi172.stdout:1.b 65 0 0 0 0 532480 0 0 65 65 active+clean 2019-12-21T15:48:50.266884+0000 46'65 48:88 [1,0,2] 1 [1,0,2] 1 0'0 2019-12-14T15:50:35.718811+0000 0'0 2019-12-21T15:48:49.587218+0000 0 2019-12-21T15:52:28.822 INFO:tasks.workunit.client.0.smithi172.stdout:2.8 66 0 0 0 0 540672 0 0 66 66 active+clean 2019-12-21T15:48:51.779146+0000 46'66 48:84 [0,2,1] 0 [0,2,1] 0 0'0 2019-12-14T15:50:36.077652+0000 0'0 2019-12-21T15:48:51.001996+0000 0 2019-12-21T15:52:28.822 INFO:tasks.workunit.client.0.smithi172.stdout:3.8 66 0 0 0 0 540672 0 0 66 66 active+clean 2019-12-21T15:48:53.596865+0000 46'66 48:81 [4,1,2] 4 [4,1,2] 4 0'0 2019-12-14T15:50:36.432117+0000 0'0 2019-12-21T15:48:52.433608+0000 0 2019-12-21T15:52:28.822 INFO:tasks.workunit.client.0.smithi172.stdout:1.a 61 0 0 0 0 499712 0 0 61 61 active+clean 2019-12-21T15:48:50.232998+0000 46'61 48:84 [1,2,4] 1 [1,2,4] 1 0'0 2019-12-14T15:50:36.788931+0000 0'0 2019-12-21T15:48:49.587218+0000 0 2019-12-21T15:52:28.822 INFO:tasks.workunit.client.0.smithi172.stdout:2.9 61 0 0 0 0 499712 0 0 61 61 active+clean 2019-12-21T15:48:51.883247+0000 46'61 48:79 [4,1,0] 4 [4,1,0] 4 0'0 2019-12-14T15:50:37.147365+0000 0'0 2019-12-21T15:48:51.001996+0000 0 2019-12-21T15:52:28.822 INFO:tasks.workunit.client.0.smithi172.stdout:3.b 65 0 0 0 0 532480 0 0 65 65 active+clean 2019-12-21T15:48:53.598249+0000 46'65 48:80 [3,0,4] 3 [3,0,4] 3 0'0 2019-12-14T15:50:37.503448+0000 0'0 2019-12-21T15:48:52.433608+0000 0 2019-12-21T15:52:28.823 INFO:tasks.workunit.client.0.smithi172.stdout:1.9 61 0 0 0 0 499712 0 0 61 61 active+clean 2019-12-21T15:48:50.190683+0000 46'61 48:84 [1,4,5] 1 [1,4,5] 1 0'0 2019-12-14T15:50:37.859855+0000 0'0 2019-12-21T15:48:49.587218+0000 0 2019-12-21T15:52:28.823 INFO:tasks.workunit.client.0.smithi172.stdout:2.a 61 0 0 0 0 499712 0 0 61 61 active+clean 2019-12-21T15:48:51.921320+0000 46'61 48:80 [1,5,0] 1 [1,5,0] 1 0'0 2019-12-14T15:50:38.219021+0000 0'0 2019-12-21T15:48:51.001996+0000 0 2019-12-21T15:52:28.823 INFO:tasks.workunit.client.0.smithi172.stdout:3.a 61 0 0 0 0 499712 0 0 61 61 active+clean 2019-12-21T15:48:53.573078+0000 46'61 48:76 [3,1,5] 3 [3,1,5] 3 0'0 2019-12-14T15:50:38.576046+0000 0'0 2019-12-21T15:48:52.433608+0000 0 2019-12-21T15:52:28.823 INFO:tasks.workunit.client.0.smithi172.stdout:1.8 66 0 0 0 0 540672 0 0 66 66 active+clean 2019-12-21T15:48:51.230728+0000 46'66 48:86 [5,4,3] 5 [5,4,3] 5 0'0 2019-12-14T15:50:38.931060+0000 0'0 2019-12-21T15:48:49.587218+0000 0 2019-12-21T15:52:28.823 INFO:tasks.workunit.client.0.smithi172.stdout:2.b 65 0 0 0 0 532480 0 0 65 65 active+clean 2019-12-21T15:48:51.922814+0000 46'65 48:85 [3,2,5] 3 [3,2,5] 3 0'0 2019-12-14T15:50:39.286910+0000 0'0 2019-12-21T15:48:51.001996+0000 0 2019-12-21T15:52:28.823 INFO:tasks.workunit.client.0.smithi172.stdout:3.5 73 0 0 0 0 598016 0 0 73 73 active+clean 2019-12-21T15:48:53.483945+0000 46'73 48:86 [5,3,4] 5 [5,3,4] 5 0'0 2019-12-14T15:50:39.644099+0000 0'0 2019-12-21T15:48:52.433608+0000 0 2019-12-21T15:52:28.824 INFO:tasks.workunit.client.0.smithi172.stdout:1.7 53 0 0 0 0 434176 0 0 53 53 active+clean 2019-12-21T15:48:50.267398+0000 46'53 48:76 [1,3,0] 1 [1,3,0] 1 0'0 2019-12-14T15:50:40.000825+0000 0'0 2019-12-21T15:48:49.587218+0000 0 2019-12-21T15:52:28.824 INFO:tasks.workunit.client.0.smithi172.stdout:2.4 51 0 0 0 0 417792 0 0 51 51 active+clean 2019-12-21T15:48:51.883549+0000 46'51 48:70 [1,0,2] 1 [1,0,2] 1 0'0 2019-12-14T15:50:40.359130+0000 0'0 2019-12-21T15:48:51.001996+0000 0 2019-12-21T15:52:28.824 INFO:tasks.workunit.client.0.smithi172.stdout:3.4 51 0 0 0 0 417792 0 0 51 51 active+clean 2019-12-21T15:48:53.596868+0000 46'51 48:66 [1,2,4] 1 [1,2,4] 1 0'0 2019-12-14T15:50:40.716191+0000 0'0 2019-12-21T15:48:52.433608+0000 0 2019-12-21T15:52:28.824 INFO:tasks.workunit.client.0.smithi172.stdout:1.6 64 0 0 0 0 524288 0 0 64 64 active+clean 2019-12-21T15:48:50.232532+0000 46'64 48:86 [3,4,2] 3 [3,4,2] 3 0'0 2019-12-14T15:50:41.074553+0000 0'0 2019-12-21T15:48:49.587218+0000 0 2019-12-21T15:52:28.824 INFO:tasks.workunit.client.0.smithi172.stdout:2.5 73 0 0 0 0 598016 0 0 73 73 active+clean 2019-12-21T15:48:51.922004+0000 46'73 48:93 [3,4,0] 3 [3,4,0] 3 0'0 2019-12-14T15:50:41.430902+0000 0'0 2019-12-21T15:48:51.001996+0000 0 2019-12-21T15:52:28.824 INFO:tasks.workunit.client.0.smithi172.stdout:1.1 67 0 0 0 0 548864 0 0 67 67 active+clean 2019-12-21T15:48:50.513172+0000 46'67 48:87 [5,0,4] 5 [5,0,4] 5 0'0 2019-12-14T15:50:41.787088+0000 0'0 2019-12-21T15:48:49.587218+0000 0 2019-12-21T15:52:28.824 INFO:tasks.workunit.client.0.smithi172.stdout:2.2 55 0 0 0 0 450560 0 0 55 55 active+clean 2019-12-21T15:48:51.882192+0000 46'55 48:71 [5,1,0] 5 [5,1,0] 5 0'0 2019-12-14T15:50:42.145860+0000 0'0 2019-12-21T15:48:51.001996+0000 0 2019-12-21T15:52:28.825 INFO:tasks.workunit.client.0.smithi172.stdout:3.3 54 0 0 0 0 442368 0 0 54 54 active+clean 2019-12-21T15:48:53.573324+0000 46'54 48:69 [4,0,5] 4 [4,0,5] 4 0'0 2019-12-14T15:50:42.500954+0000 0'0 2019-12-21T15:48:52.433608+0000 0 2019-12-21T15:52:28.825 INFO:tasks.workunit.client.0.smithi172.stdout:1.0 72 0 0 0 0 589824 0 0 72 72 active+clean 2019-12-21T15:48:50.266436+0000 46'72 48:95 [1,0,2] 1 [1,0,2] 1 0'0 2019-12-14T15:50:42.876490+0000 0'0 2019-12-21T15:48:49.587218+0000 0 2019-12-21T15:52:28.826 INFO:tasks.workunit.client.0.smithi172.stdout:2.3 54 0 0 0 0 442368 0 0 54 54 active+clean 2019-12-21T15:48:51.921328+0000 46'54 48:70 [5,4,1] 5 [5,4,1] 5 0'0 2019-12-14T15:50:43.233249+0000 0'0 2019-12-21T15:48:51.001996+0000 0 2019-12-21T15:52:28.826 INFO:tasks.workunit.client.0.smithi172.stdout:3.2 55 0 0 0 0 450560 0 0 55 55 active+clean 2019-12-21T15:48:53.597372+0000 46'55 48:70 [3,5,0] 3 [3,5,0] 3 0'0 2019-12-14T15:50:43.589419+0000 0'0 2019-12-21T15:48:52.433608+0000 0 2019-12-21T15:52:28.826 INFO:tasks.workunit.client.0.smithi172.stdout:1.2 55 0 0 0 0 450560 0 0 55 55 active+clean 2019-12-21T15:48:50.266115+0000 46'55 48:77 [0,4,1] 0 [0,4,1] 0 0'0 2019-12-14T15:50:43.946014+0000 0'0 2019-12-21T15:48:49.587218+0000 0 2019-12-21T15:52:28.826 INFO:tasks.workunit.client.0.smithi172.stdout:2.1 67 0 0 0 0 548864 0 0 67 67 active+clean 2019-12-21T15:48:51.921897+0000 46'67 46:83 [2,5,0] 2 [2,5,0] 2 0'0 2019-12-21T15:48:51.001996+0000 0'0 2019-12-21T15:48:51.001996+0000 0 2019-12-21T15:52:28.826 INFO:tasks.workunit.client.0.smithi172.stdout:3.0 72 0 0 0 0 589824 0 0 72 72 active+clean 2019-12-21T15:48:53.598105+0000 46'72 48:87 [1,2,4] 1 [1,2,4] 1 0'0 2019-12-14T15:50:44.658995+0000 0'0 2019-12-21T15:48:52.433608+0000 0 2019-12-21T15:52:28.826 INFO:tasks.workunit.client.0.smithi172.stdout:1.3 54 0 0 0 0 442368 0 0 54 54 active+clean 2019-12-21T15:48:50.232614+0000 46'54 48:75 [4,1,2] 4 [4,1,2] 4 0'0 2019-12-14T15:50:45.014718+0000 0'0 2019-12-21T15:48:49.587218+0000 0 2019-12-21T15:52:28.826 INFO:tasks.workunit.client.0.smithi172.stdout:2.0 72 0 0 0 0 589824 0 0 72 72 active+clean 2019-12-21T15:48:51.921796+0000 46'72 48:92 [3,5,1] 3 [3,5,1] 3 0'0 2019-12-14T15:50:45.372506+0000 0'0 2019-12-21T15:48:51.001996+0000 0 2019-12-21T15:52:28.827 INFO:tasks.workunit.client.0.smithi172.stdout:3.1 67 0 0 0 0 548864 0 0 67 67 active+clean 2019-12-21T15:48:53.483350+0000 46'67 48:82 [4,3,5] 4 [4,3,5] 4 0'0 2019-12-14T15:50:45.728301+0000 0'0 2019-12-21T15:48:52.433608+0000 0 2019-12-21T15:52:28.827 INFO:tasks.workunit.client.0.smithi172.stdout:3.6 64 0 0 0 0 524288 0 0 64 64 active+clean 2019-12-21T15:48:53.596770+0000 46'64 47:77 [0,1,4] 0 [0,1,4] 0 0'0 2019-12-21T15:48:52.433608+0000 0'0 2019-12-21T15:48:52.433608+0000 0 2019-12-21T15:52:28.827 INFO:tasks.workunit.client.0.smithi172.stdout:1.4 51 0 0 0 0 417792 0 0 51 51 active+clean 2019-12-21T15:48:50.267005+0000 46'51 47:72 [3,0,1] 3 [3,0,1] 3 0'0 2019-12-21T15:48:49.587218+0000 0'0 2019-12-21T15:48:49.587218+0000 0 2019-12-21T15:52:28.827 INFO:tasks.workunit.client.0.smithi172.stdout:2.7 53 0 0 0 0 434176 0 0 53 53 active+clean 2019-12-21T15:48:51.882809+0000 46'53 47:72 [3,0,2] 3 [3,0,2] 3 0'0 2019-12-21T15:48:51.001996+0000 0'0 2019-12-21T15:48:51.001996+0000 0 2019-12-21T15:52:28.827 INFO:tasks.workunit.client.0.smithi172.stdout:3.7 53 0 0 0 0 434176 0 0 53 53 active+clean 2019-12-21T15:48:53.573718+0000 46'53 48:68 [4,3,0] 4 [4,3,0] 4 0'0 2019-12-14T15:50:47.156376+0000 0'0 2019-12-21T15:48:52.433608+0000 0 2019-12-21T15:52:28.827 INFO:tasks.workunit.client.0.smithi172.stdout:1.5 73 0 0 0 0 598016 0 0 73 73 active+clean 2019-12-21T15:48:50.232808+0000 46'73 47:93 [4,2,5] 4 [4,2,5] 4 0'0 2019-12-21T15:48:49.587218+0000 0'0 2019-12-21T15:48:49.587218+0000 0 2019-12-21T15:52:28.827 INFO:tasks.workunit.client.0.smithi172.stdout:2.6 64 0 0 0 0 524288 0 0 64 64 active+clean 2019-12-21T15:48:51.921603+0000 46'64 47:82 [1,0,4] 1 [1,0,4] 1 0'0 2019-12-21T15:48:51.001996+0000 0'0 2019-12-21T15:48:51.001996+0000 0 2019-12-21T15:52:28.828 INFO:tasks.workunit.client.0.smithi172.stdout: 2019-12-21T15:52:28.828 INFO:tasks.workunit.client.0.smithi172.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilisation. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2019-12-21T15:52:28.828 INFO:tasks.workunit.client.0.smithi172.stderr:dumped pgs 2019-12-21T15:52:28.835 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:106: TEST_recover_unexpected: ceph pg dump pgs 2019-12-21T15:52:28.836 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:106: TEST_recover_unexpected: grep scrubbing 2019-12-21T15:52:29.051 INFO:tasks.workunit.client.0.smithi172.stderr:dumped pgs 2019-12-21T15:52:29.059 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:110: TEST_recover_unexpected: sleep 1 2019-12-21T15:52:30.060 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:103: TEST_recover_unexpected: for i in $(seq 0 300) 2019-12-21T15:52:30.060 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:105: TEST_recover_unexpected: ceph pg dump pgs 2019-12-21T15:52:30.283 INFO:tasks.workunit.client.0.smithi172.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2019-12-21T15:52:30.283 INFO:tasks.workunit.client.0.smithi172.stdout:3.d 67 0 0 0 0 548864 0 0 67 67 active+clean 2019-12-21T15:48:53.596654+0000 46'67 48:82 [3,1,0] 3 [3,1,0] 3 0'0 2019-12-14T15:50:31.047070+0000 0'0 2019-12-21T15:48:52.433608+0000 0 2019-12-21T15:52:30.283 INFO:tasks.workunit.client.0.smithi172.stdout:1.f 62 0 0 0 0 507904 0 0 62 62 active+clean 2019-12-21T15:48:50.267215+0000 46'62 48:84 [3,0,4] 3 [3,0,4] 3 0'0 2019-12-14T15:50:31.406387+0000 0'0 2019-12-21T15:48:49.587218+0000 0 2019-12-21T15:52:30.283 INFO:tasks.workunit.client.0.smithi172.stdout:2.c 60 0 0 0 0 491520 0 0 60 60 active+clean 2019-12-21T15:48:51.921595+0000 46'60 48:77 [2,0,5] 2 [2,0,5] 2 0'0 2019-12-14T15:50:31.765715+0000 0'0 2019-12-21T15:48:51.001996+0000 0 2019-12-21T15:52:30.283 INFO:tasks.workunit.client.0.smithi172.stdout:3.c 60 0 0 0 0 491520 0 0 60 60 active+clean 2019-12-21T15:48:53.573137+0000 46'60 48:73 [5,3,1] 5 [5,3,1] 5 0'0 2019-12-14T15:50:32.121892+0000 0'0 2019-12-21T15:48:52.433608+0000 0 2019-12-21T15:52:30.284 INFO:tasks.workunit.client.0.smithi172.stdout:1.e 69 0 0 0 0 565248 0 0 69 69 active+clean 2019-12-21T15:48:50.233389+0000 46'69 48:92 [1,4,2] 1 [1,4,2] 1 0'0 2019-12-14T15:50:32.477832+0000 0'0 2019-12-21T15:48:49.587218+0000 0 2019-12-21T15:52:30.284 INFO:tasks.workunit.client.0.smithi172.stdout:2.d 67 0 0 0 0 548864 0 0 67 67 active+clean 2019-12-21T15:48:51.831821+0000 46'67 48:83 [5,1,3] 5 [5,1,3] 5 0'0 2019-12-14T15:50:32.835272+0000 0'0 2019-12-21T15:48:51.001996+0000 0 2019-12-21T15:52:30.284 INFO:tasks.workunit.client.0.smithi172.stdout:3.f 62 0 0 0 0 507904 0 0 62 62 active+clean 2019-12-21T15:48:53.597626+0000 46'62 48:76 [0,1,4] 0 [0,1,4] 0 0'0 2019-12-14T15:50:33.192542+0000 0'0 2019-12-21T15:48:52.433608+0000 0 2019-12-21T15:52:30.284 INFO:tasks.workunit.client.0.smithi172.stdout:1.d 67 0 0 0 0 548864 0 0 67 67 active+clean 2019-12-21T15:48:50.232330+0000 46'67 48:87 [2,4,1] 2 [2,4,1] 2 0'0 2019-12-14T15:50:33.550430+0000 0'0 2019-12-21T15:48:49.587218+0000 0 2019-12-21T15:52:30.284 INFO:tasks.workunit.client.0.smithi172.stdout:2.e 69 0 0 0 0 565248 0 0 69 69 active+clean 2019-12-21T15:48:51.921425+0000 46'69 48:86 [2,3,5] 2 [2,3,5] 2 0'0 2019-12-14T15:50:33.906226+0000 0'0 2019-12-21T15:48:51.001996+0000 0 2019-12-21T15:52:30.284 INFO:tasks.workunit.client.0.smithi172.stdout:3.e 69 0 0 0 0 565248 0 0 69 69 active+clean 2019-12-21T15:48:53.597334+0000 46'69 48:82 [2,4,1] 2 [2,4,1] 2 0'0 2019-12-14T15:50:34.263496+0000 0'0 2019-12-21T15:48:52.433608+0000 0 2019-12-21T15:52:30.284 INFO:tasks.workunit.client.0.smithi172.stdout:1.c 60 0 0 0 0 491520 0 0 60 60 active+clean 2019-12-21T15:48:50.266642+0000 46'60 48:82 [3,0,4] 3 [3,0,4] 3 0'0 2019-12-14T15:50:34.647688+0000 0'0 2019-12-21T15:48:49.587218+0000 0 2019-12-21T15:52:30.285 INFO:tasks.workunit.client.0.smithi172.stdout:2.f 62 0 0 0 0 507904 0 0 62 62 active+clean 2019-12-21T15:48:51.780390+0000 46'62 48:80 [0,5,4] 0 [0,5,4] 0 0'0 2019-12-14T15:50:35.003467+0000 0'0 2019-12-21T15:48:51.001996+0000 0 2019-12-21T15:52:30.285 INFO:tasks.workunit.client.0.smithi172.stdout:3.9 61 0 0 0 0 499712 0 0 61 61 active+clean 2019-12-21T15:48:53.597754+0000 46'61 48:76 [4,2,5] 4 [4,2,5] 4 0'0 2019-12-14T15:50:35.361209+0000 0'0 2019-12-21T15:48:52.433608+0000 0 2019-12-21T15:52:30.285 INFO:tasks.workunit.client.0.smithi172.stdout:1.b 65 0 0 0 0 532480 0 0 65 65 active+clean 2019-12-21T15:48:50.266884+0000 46'65 48:88 [1,0,2] 1 [1,0,2] 1 0'0 2019-12-14T15:50:35.718811+0000 0'0 2019-12-21T15:48:49.587218+0000 0 2019-12-21T15:52:30.285 INFO:tasks.workunit.client.0.smithi172.stdout:2.8 66 0 0 0 0 540672 0 0 66 66 active+clean 2019-12-21T15:48:51.779146+0000 46'66 48:84 [0,2,1] 0 [0,2,1] 0 0'0 2019-12-14T15:50:36.077652+0000 0'0 2019-12-21T15:48:51.001996+0000 0 2019-12-21T15:52:30.285 INFO:tasks.workunit.client.0.smithi172.stdout:3.8 66 0 0 0 0 540672 0 0 66 66 active+clean 2019-12-21T15:48:53.596865+0000 46'66 48:81 [4,1,2] 4 [4,1,2] 4 0'0 2019-12-14T15:50:36.432117+0000 0'0 2019-12-21T15:48:52.433608+0000 0 2019-12-21T15:52:30.285 INFO:tasks.workunit.client.0.smithi172.stdout:1.a 61 0 0 0 0 499712 0 0 61 61 active+clean 2019-12-21T15:48:50.232998+0000 46'61 48:84 [1,2,4] 1 [1,2,4] 1 0'0 2019-12-14T15:50:36.788931+0000 0'0 2019-12-21T15:48:49.587218+0000 0 2019-12-21T15:52:30.286 INFO:tasks.workunit.client.0.smithi172.stdout:2.9 61 0 0 0 0 499712 0 0 61 61 active+clean 2019-12-21T15:48:51.883247+0000 46'61 48:79 [4,1,0] 4 [4,1,0] 4 0'0 2019-12-14T15:50:37.147365+0000 0'0 2019-12-21T15:48:51.001996+0000 0 2019-12-21T15:52:30.286 INFO:tasks.workunit.client.0.smithi172.stdout:3.b 65 0 0 0 0 532480 0 0 65 65 active+clean 2019-12-21T15:48:53.598249+0000 46'65 48:80 [3,0,4] 3 [3,0,4] 3 0'0 2019-12-14T15:50:37.503448+0000 0'0 2019-12-21T15:48:52.433608+0000 0 2019-12-21T15:52:30.286 INFO:tasks.workunit.client.0.smithi172.stdout:1.9 61 0 0 0 0 499712 0 0 61 61 active+clean 2019-12-21T15:48:50.190683+0000 46'61 48:84 [1,4,5] 1 [1,4,5] 1 0'0 2019-12-14T15:50:37.859855+0000 0'0 2019-12-21T15:48:49.587218+0000 0 2019-12-21T15:52:30.286 INFO:tasks.workunit.client.0.smithi172.stdout:2.a 61 0 0 0 0 499712 0 0 61 61 active+clean 2019-12-21T15:48:51.921320+0000 46'61 48:80 [1,5,0] 1 [1,5,0] 1 0'0 2019-12-14T15:50:38.219021+0000 0'0 2019-12-21T15:48:51.001996+0000 0 2019-12-21T15:52:30.286 INFO:tasks.workunit.client.0.smithi172.stdout:3.a 61 0 0 0 0 499712 0 0 61 61 active+clean 2019-12-21T15:48:53.573078+0000 46'61 48:76 [3,1,5] 3 [3,1,5] 3 0'0 2019-12-14T15:50:38.576046+0000 0'0 2019-12-21T15:48:52.433608+0000 0 2019-12-21T15:52:30.286 INFO:tasks.workunit.client.0.smithi172.stdout:1.8 66 0 0 0 0 540672 0 0 66 66 active+clean 2019-12-21T15:48:51.230728+0000 46'66 48:86 [5,4,3] 5 [5,4,3] 5 0'0 2019-12-14T15:50:38.931060+0000 0'0 2019-12-21T15:48:49.587218+0000 0 2019-12-21T15:52:30.286 INFO:tasks.workunit.client.0.smithi172.stdout:2.b 65 0 0 0 0 532480 0 0 65 65 active+clean 2019-12-21T15:48:51.922814+0000 46'65 48:85 [3,2,5] 3 [3,2,5] 3 0'0 2019-12-14T15:50:39.286910+0000 0'0 2019-12-21T15:48:51.001996+0000 0 2019-12-21T15:52:30.287 INFO:tasks.workunit.client.0.smithi172.stdout:3.5 73 0 0 0 0 598016 0 0 73 73 active+clean 2019-12-21T15:48:53.483945+0000 46'73 48:86 [5,3,4] 5 [5,3,4] 5 0'0 2019-12-14T15:50:39.644099+0000 0'0 2019-12-21T15:48:52.433608+0000 0 2019-12-21T15:52:30.287 INFO:tasks.workunit.client.0.smithi172.stdout:1.7 53 0 0 0 0 434176 0 0 53 53 active+clean 2019-12-21T15:48:50.267398+0000 46'53 48:76 [1,3,0] 1 [1,3,0] 1 0'0 2019-12-14T15:50:40.000825+0000 0'0 2019-12-21T15:48:49.587218+0000 0 2019-12-21T15:52:30.287 INFO:tasks.workunit.client.0.smithi172.stdout:2.4 51 0 0 0 0 417792 0 0 51 51 active+clean 2019-12-21T15:48:51.883549+0000 46'51 48:70 [1,0,2] 1 [1,0,2] 1 0'0 2019-12-14T15:50:40.359130+0000 0'0 2019-12-21T15:48:51.001996+0000 0 2019-12-21T15:52:30.287 INFO:tasks.workunit.client.0.smithi172.stdout:3.4 51 0 0 0 0 417792 0 0 51 51 active+clean 2019-12-21T15:48:53.596868+0000 46'51 48:66 [1,2,4] 1 [1,2,4] 1 0'0 2019-12-14T15:50:40.716191+0000 0'0 2019-12-21T15:48:52.433608+0000 0 2019-12-21T15:52:30.287 INFO:tasks.workunit.client.0.smithi172.stdout:1.6 64 0 0 0 0 524288 0 0 64 64 active+clean 2019-12-21T15:48:50.232532+0000 46'64 48:86 [3,4,2] 3 [3,4,2] 3 0'0 2019-12-14T15:50:41.074553+0000 0'0 2019-12-21T15:48:49.587218+0000 0 2019-12-21T15:52:30.287 INFO:tasks.workunit.client.0.smithi172.stdout:2.5 73 0 0 0 0 598016 0 0 73 73 active+clean 2019-12-21T15:48:51.922004+0000 46'73 48:93 [3,4,0] 3 [3,4,0] 3 0'0 2019-12-14T15:50:41.430902+0000 0'0 2019-12-21T15:48:51.001996+0000 0 2019-12-21T15:52:30.287 INFO:tasks.workunit.client.0.smithi172.stdout:1.1 67 0 0 0 0 548864 0 0 67 67 active+clean 2019-12-21T15:48:50.513172+0000 46'67 48:87 [5,0,4] 5 [5,0,4] 5 0'0 2019-12-14T15:50:41.787088+0000 0'0 2019-12-21T15:48:49.587218+0000 0 2019-12-21T15:52:30.288 INFO:tasks.workunit.client.0.smithi172.stdout:2.2 55 0 0 0 0 450560 0 0 55 55 active+clean 2019-12-21T15:48:51.882192+0000 46'55 48:71 [5,1,0] 5 [5,1,0] 5 0'0 2019-12-14T15:50:42.145860+0000 0'0 2019-12-21T15:48:51.001996+0000 0 2019-12-21T15:52:30.288 INFO:tasks.workunit.client.0.smithi172.stdout:3.3 54 0 0 0 0 442368 0 0 54 54 active+clean 2019-12-21T15:48:53.573324+0000 46'54 48:69 [4,0,5] 4 [4,0,5] 4 0'0 2019-12-14T15:50:42.500954+0000 0'0 2019-12-21T15:48:52.433608+0000 0 2019-12-21T15:52:30.288 INFO:tasks.workunit.client.0.smithi172.stdout:1.0 72 0 0 0 0 589824 0 0 72 72 active+clean 2019-12-21T15:48:50.266436+0000 46'72 48:95 [1,0,2] 1 [1,0,2] 1 0'0 2019-12-14T15:50:42.876490+0000 0'0 2019-12-21T15:48:49.587218+0000 0 2019-12-21T15:52:30.288 INFO:tasks.workunit.client.0.smithi172.stdout:2.3 54 0 0 0 0 442368 0 0 54 54 active+clean 2019-12-21T15:48:51.921328+0000 46'54 48:70 [5,4,1] 5 [5,4,1] 5 0'0 2019-12-14T15:50:43.233249+0000 0'0 2019-12-21T15:48:51.001996+0000 0 2019-12-21T15:52:30.288 INFO:tasks.workunit.client.0.smithi172.stdout:3.2 55 0 0 0 0 450560 0 0 55 55 active+clean 2019-12-21T15:48:53.597372+0000 46'55 48:70 [3,5,0] 3 [3,5,0] 3 0'0 2019-12-14T15:50:43.589419+0000 0'0 2019-12-21T15:48:52.433608+0000 0 2019-12-21T15:52:30.288 INFO:tasks.workunit.client.0.smithi172.stdout:1.2 55 0 0 0 0 450560 0 0 55 55 active+clean 2019-12-21T15:48:50.266115+0000 46'55 48:77 [0,4,1] 0 [0,4,1] 0 0'0 2019-12-14T15:50:43.946014+0000 0'0 2019-12-21T15:48:49.587218+0000 0 2019-12-21T15:52:30.288 INFO:tasks.workunit.client.0.smithi172.stdout:2.1 67 0 0 0 0 548864 0 0 67 67 active+clean 2019-12-21T15:48:51.921897+0000 46'67 46:83 [2,5,0] 2 [2,5,0] 2 0'0 2019-12-21T15:48:51.001996+0000 0'0 2019-12-21T15:48:51.001996+0000 0 2019-12-21T15:52:30.289 INFO:tasks.workunit.client.0.smithi172.stdout:3.0 72 0 0 0 0 589824 0 0 72 72 active+clean 2019-12-21T15:48:53.598105+0000 46'72 48:87 [1,2,4] 1 [1,2,4] 1 0'0 2019-12-14T15:50:44.658995+0000 0'0 2019-12-21T15:48:52.433608+0000 0 2019-12-21T15:52:30.289 INFO:tasks.workunit.client.0.smithi172.stdout:1.3 54 0 0 0 0 442368 0 0 54 54 active+clean 2019-12-21T15:48:50.232614+0000 46'54 48:75 [4,1,2] 4 [4,1,2] 4 0'0 2019-12-14T15:50:45.014718+0000 0'0 2019-12-21T15:48:49.587218+0000 0 2019-12-21T15:52:30.289 INFO:tasks.workunit.client.0.smithi172.stdout:2.0 72 0 0 0 0 589824 0 0 72 72 active+clean 2019-12-21T15:48:51.921796+0000 46'72 48:92 [3,5,1] 3 [3,5,1] 3 0'0 2019-12-14T15:50:45.372506+0000 0'0 2019-12-21T15:48:51.001996+0000 0 2019-12-21T15:52:30.289 INFO:tasks.workunit.client.0.smithi172.stdout:3.1 67 0 0 0 0 548864 0 0 67 67 active+clean 2019-12-21T15:48:53.483350+0000 46'67 48:82 [4,3,5] 4 [4,3,5] 4 0'0 2019-12-14T15:50:45.728301+0000 0'0 2019-12-21T15:48:52.433608+0000 0 2019-12-21T15:52:30.289 INFO:tasks.workunit.client.0.smithi172.stdout:3.6 64 0 0 0 0 524288 0 0 64 64 active+clean 2019-12-21T15:48:53.596770+0000 46'64 47:77 [0,1,4] 0 [0,1,4] 0 0'0 2019-12-21T15:48:52.433608+0000 0'0 2019-12-21T15:48:52.433608+0000 0 2019-12-21T15:52:30.289 INFO:tasks.workunit.client.0.smithi172.stdout:1.4 51 0 0 0 0 417792 0 0 51 51 active+clean 2019-12-21T15:48:50.267005+0000 46'51 47:72 [3,0,1] 3 [3,0,1] 3 0'0 2019-12-21T15:48:49.587218+0000 0'0 2019-12-21T15:48:49.587218+0000 0 2019-12-21T15:52:30.289 INFO:tasks.workunit.client.0.smithi172.stdout:2.7 53 0 0 0 0 434176 0 0 53 53 active+clean 2019-12-21T15:48:51.882809+0000 46'53 47:72 [3,0,2] 3 [3,0,2] 3 0'0 2019-12-21T15:48:51.001996+0000 0'0 2019-12-21T15:48:51.001996+0000 0 2019-12-21T15:52:30.290 INFO:tasks.workunit.client.0.smithi172.stdout:3.7 53 0 0 0 0 434176 0 0 53 53 active+clean 2019-12-21T15:48:53.573718+0000 46'53 48:68 [4,3,0] 4 [4,3,0] 4 0'0 2019-12-14T15:50:47.156376+0000 0'0 2019-12-21T15:48:52.433608+0000 0 2019-12-21T15:52:30.290 INFO:tasks.workunit.client.0.smithi172.stdout:1.5 73 0 0 0 0 598016 0 0 73 73 active+clean 2019-12-21T15:48:50.232808+0000 46'73 47:93 [4,2,5] 4 [4,2,5] 4 0'0 2019-12-21T15:48:49.587218+0000 0'0 2019-12-21T15:48:49.587218+0000 0 2019-12-21T15:52:30.290 INFO:tasks.workunit.client.0.smithi172.stdout:2.6 64 0 0 0 0 524288 0 0 64 64 active+clean 2019-12-21T15:48:51.921603+0000 46'64 47:82 [1,0,4] 1 [1,0,4] 1 0'0 2019-12-21T15:48:51.001996+0000 0'0 2019-12-21T15:48:51.001996+0000 0 2019-12-21T15:52:30.290 INFO:tasks.workunit.client.0.smithi172.stdout: 2019-12-21T15:52:30.300 INFO:tasks.workunit.client.0.smithi172.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilisation. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2019-12-21T15:52:30.301 INFO:tasks.workunit.client.0.smithi172.stderr:dumped pgs 2019-12-21T15:52:30.301 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:106: TEST_recover_unexpected: ceph pg dump pgs 2019-12-21T15:52:30.302 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:106: TEST_recover_unexpected: grep scrubbing 2019-12-21T15:52:30.514 INFO:tasks.workunit.client.0.smithi172.stderr:dumped pgs 2019-12-21T15:52:30.526 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:110: TEST_recover_unexpected: sleep 1 2019-12-21T15:52:31.523 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:103: TEST_recover_unexpected: for i in $(seq 0 300) 2019-12-21T15:52:31.523 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:105: TEST_recover_unexpected: ceph pg dump pgs 2019-12-21T15:52:31.755 INFO:tasks.workunit.client.0.smithi172.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2019-12-21T15:52:31.755 INFO:tasks.workunit.client.0.smithi172.stdout:3.d 67 0 0 0 0 548864 0 0 67 67 active+clean 2019-12-21T15:48:53.596654+0000 46'67 48:83 [3,1,0] 3 [3,1,0] 3 0'0 2019-12-14T15:50:31.047070+0000 0'0 2019-12-21T15:48:52.433608+0000 0 2019-12-21T15:52:31.756 INFO:tasks.workunit.client.0.smithi172.stdout:1.f 62 0 0 0 0 507904 0 0 62 62 active+clean 2019-12-21T15:48:50.267215+0000 46'62 48:85 [3,0,4] 3 [3,0,4] 3 0'0 2019-12-14T15:50:31.406387+0000 0'0 2019-12-21T15:48:49.587218+0000 0 2019-12-21T15:52:31.756 INFO:tasks.workunit.client.0.smithi172.stdout:2.c 60 0 0 0 0 491520 0 0 60 60 active+clean 2019-12-21T15:48:51.921595+0000 46'60 48:78 [2,0,5] 2 [2,0,5] 2 0'0 2019-12-14T15:50:31.765715+0000 0'0 2019-12-21T15:48:51.001996+0000 0 2019-12-21T15:52:31.756 INFO:tasks.workunit.client.0.smithi172.stdout:3.c 60 0 0 0 0 491520 0 0 60 60 active+clean 2019-12-21T15:48:53.573137+0000 46'60 48:73 [5,3,1] 5 [5,3,1] 5 0'0 2019-12-14T15:50:32.121892+0000 0'0 2019-12-21T15:48:52.433608+0000 0 2019-12-21T15:52:31.756 INFO:tasks.workunit.client.0.smithi172.stdout:1.e 69 0 0 0 0 565248 0 0 69 69 active+clean 2019-12-21T15:48:50.233389+0000 46'69 48:92 [1,4,2] 1 [1,4,2] 1 0'0 2019-12-14T15:50:32.477832+0000 0'0 2019-12-21T15:48:49.587218+0000 0 2019-12-21T15:52:31.756 INFO:tasks.workunit.client.0.smithi172.stdout:2.d 67 0 0 0 0 548864 0 0 67 67 active+clean 2019-12-21T15:48:51.831821+0000 46'67 48:83 [5,1,3] 5 [5,1,3] 5 0'0 2019-12-14T15:50:32.835272+0000 0'0 2019-12-21T15:48:51.001996+0000 0 2019-12-21T15:52:31.756 INFO:tasks.workunit.client.0.smithi172.stdout:3.f 62 0 0 0 0 507904 0 0 62 62 active+clean 2019-12-21T15:48:53.597626+0000 46'62 48:77 [0,1,4] 0 [0,1,4] 0 0'0 2019-12-14T15:50:33.192542+0000 0'0 2019-12-21T15:48:52.433608+0000 0 2019-12-21T15:52:31.757 INFO:tasks.workunit.client.0.smithi172.stdout:1.d 67 0 0 0 0 548864 0 0 67 67 active+clean 2019-12-21T15:48:50.232330+0000 46'67 48:88 [2,4,1] 2 [2,4,1] 2 0'0 2019-12-14T15:50:33.550430+0000 0'0 2019-12-21T15:48:49.587218+0000 0 2019-12-21T15:52:31.757 INFO:tasks.workunit.client.0.smithi172.stdout:2.e 69 0 0 0 0 565248 0 0 69 69 active+clean 2019-12-21T15:48:51.921425+0000 46'69 48:87 [2,3,5] 2 [2,3,5] 2 0'0 2019-12-14T15:50:33.906226+0000 0'0 2019-12-21T15:48:51.001996+0000 0 2019-12-21T15:52:31.757 INFO:tasks.workunit.client.0.smithi172.stdout:3.e 69 0 0 0 0 565248 0 0 69 69 active+clean 2019-12-21T15:48:53.597334+0000 46'69 48:83 [2,4,1] 2 [2,4,1] 2 0'0 2019-12-14T15:50:34.263496+0000 0'0 2019-12-21T15:48:52.433608+0000 0 2019-12-21T15:52:31.757 INFO:tasks.workunit.client.0.smithi172.stdout:1.c 60 0 0 0 0 491520 0 0 60 60 active+clean 2019-12-21T15:48:50.266642+0000 46'60 48:83 [3,0,4] 3 [3,0,4] 3 0'0 2019-12-14T15:50:34.647688+0000 0'0 2019-12-21T15:48:49.587218+0000 0 2019-12-21T15:52:31.757 INFO:tasks.workunit.client.0.smithi172.stdout:2.f 62 0 0 0 0 507904 0 0 62 62 active+clean 2019-12-21T15:48:51.780390+0000 46'62 48:81 [0,5,4] 0 [0,5,4] 0 0'0 2019-12-14T15:50:35.003467+0000 0'0 2019-12-21T15:48:51.001996+0000 0 2019-12-21T15:52:31.757 INFO:tasks.workunit.client.0.smithi172.stdout:3.9 61 0 0 0 0 499712 0 0 61 61 active+clean 2019-12-21T15:48:53.597754+0000 46'61 48:76 [4,2,5] 4 [4,2,5] 4 0'0 2019-12-14T15:50:35.361209+0000 0'0 2019-12-21T15:48:52.433608+0000 0 2019-12-21T15:52:31.757 INFO:tasks.workunit.client.0.smithi172.stdout:1.b 65 0 0 0 0 532480 0 0 65 65 active+clean 2019-12-21T15:48:50.266884+0000 46'65 48:88 [1,0,2] 1 [1,0,2] 1 0'0 2019-12-14T15:50:35.718811+0000 0'0 2019-12-21T15:48:49.587218+0000 0 2019-12-21T15:52:31.758 INFO:tasks.workunit.client.0.smithi172.stdout:2.8 66 0 0 0 0 540672 0 0 66 66 active+clean 2019-12-21T15:48:51.779146+0000 46'66 48:85 [0,2,1] 0 [0,2,1] 0 0'0 2019-12-14T15:50:36.077652+0000 0'0 2019-12-21T15:48:51.001996+0000 0 2019-12-21T15:52:31.758 INFO:tasks.workunit.client.0.smithi172.stdout:3.8 66 0 0 0 0 540672 0 0 66 66 active+clean 2019-12-21T15:48:53.596865+0000 46'66 48:81 [4,1,2] 4 [4,1,2] 4 0'0 2019-12-14T15:50:36.432117+0000 0'0 2019-12-21T15:48:52.433608+0000 0 2019-12-21T15:52:31.758 INFO:tasks.workunit.client.0.smithi172.stdout:1.a 61 0 0 0 0 499712 0 0 61 61 active+clean 2019-12-21T15:48:50.232998+0000 46'61 48:84 [1,2,4] 1 [1,2,4] 1 0'0 2019-12-14T15:50:36.788931+0000 0'0 2019-12-21T15:48:49.587218+0000 0 2019-12-21T15:52:31.758 INFO:tasks.workunit.client.0.smithi172.stdout:2.9 61 0 0 0 0 499712 0 0 61 61 active+clean 2019-12-21T15:48:51.883247+0000 46'61 48:79 [4,1,0] 4 [4,1,0] 4 0'0 2019-12-14T15:50:37.147365+0000 0'0 2019-12-21T15:48:51.001996+0000 0 2019-12-21T15:52:31.758 INFO:tasks.workunit.client.0.smithi172.stdout:3.b 65 0 0 0 0 532480 0 0 65 65 active+clean 2019-12-21T15:48:53.598249+0000 46'65 48:81 [3,0,4] 3 [3,0,4] 3 0'0 2019-12-14T15:50:37.503448+0000 0'0 2019-12-21T15:48:52.433608+0000 0 2019-12-21T15:52:31.758 INFO:tasks.workunit.client.0.smithi172.stdout:1.9 61 0 0 0 0 499712 0 0 61 61 active+clean 2019-12-21T15:48:50.190683+0000 46'61 48:84 [1,4,5] 1 [1,4,5] 1 0'0 2019-12-14T15:50:37.859855+0000 0'0 2019-12-21T15:48:49.587218+0000 0 2019-12-21T15:52:31.758 INFO:tasks.workunit.client.0.smithi172.stdout:2.a 61 0 0 0 0 499712 0 0 61 61 active+clean 2019-12-21T15:48:51.921320+0000 46'61 48:80 [1,5,0] 1 [1,5,0] 1 0'0 2019-12-14T15:50:38.219021+0000 0'0 2019-12-21T15:48:51.001996+0000 0 2019-12-21T15:52:31.759 INFO:tasks.workunit.client.0.smithi172.stdout:3.a 61 0 0 0 0 499712 0 0 61 61 active+clean 2019-12-21T15:48:53.573078+0000 46'61 48:77 [3,1,5] 3 [3,1,5] 3 0'0 2019-12-14T15:50:38.576046+0000 0'0 2019-12-21T15:48:52.433608+0000 0 2019-12-21T15:52:31.759 INFO:tasks.workunit.client.0.smithi172.stdout:1.8 66 0 0 0 0 540672 0 0 66 66 active+clean 2019-12-21T15:48:51.230728+0000 46'66 48:86 [5,4,3] 5 [5,4,3] 5 0'0 2019-12-14T15:50:38.931060+0000 0'0 2019-12-21T15:48:49.587218+0000 0 2019-12-21T15:52:31.759 INFO:tasks.workunit.client.0.smithi172.stdout:2.b 65 0 0 0 0 532480 0 0 65 65 active+clean 2019-12-21T15:48:51.922814+0000 46'65 48:86 [3,2,5] 3 [3,2,5] 3 0'0 2019-12-14T15:50:39.286910+0000 0'0 2019-12-21T15:48:51.001996+0000 0 2019-12-21T15:52:31.759 INFO:tasks.workunit.client.0.smithi172.stdout:3.5 73 0 0 0 0 598016 0 0 73 73 active+clean 2019-12-21T15:48:53.483945+0000 46'73 48:86 [5,3,4] 5 [5,3,4] 5 0'0 2019-12-14T15:50:39.644099+0000 0'0 2019-12-21T15:48:52.433608+0000 0 2019-12-21T15:52:31.759 INFO:tasks.workunit.client.0.smithi172.stdout:1.7 53 0 0 0 0 434176 0 0 53 53 active+clean 2019-12-21T15:48:50.267398+0000 46'53 48:76 [1,3,0] 1 [1,3,0] 1 0'0 2019-12-14T15:50:40.000825+0000 0'0 2019-12-21T15:48:49.587218+0000 0 2019-12-21T15:52:31.759 INFO:tasks.workunit.client.0.smithi172.stdout:2.4 51 0 0 0 0 417792 0 0 51 51 active+clean 2019-12-21T15:48:51.883549+0000 46'51 48:70 [1,0,2] 1 [1,0,2] 1 0'0 2019-12-14T15:50:40.359130+0000 0'0 2019-12-21T15:48:51.001996+0000 0 2019-12-21T15:52:31.759 INFO:tasks.workunit.client.0.smithi172.stdout:3.4 51 0 0 0 0 417792 0 0 51 51 active+clean 2019-12-21T15:48:53.596868+0000 46'51 48:66 [1,2,4] 1 [1,2,4] 1 0'0 2019-12-14T15:50:40.716191+0000 0'0 2019-12-21T15:48:52.433608+0000 0 2019-12-21T15:52:31.760 INFO:tasks.workunit.client.0.smithi172.stdout:1.6 64 0 0 0 0 524288 0 0 64 64 active+clean 2019-12-21T15:48:50.232532+0000 46'64 48:87 [3,4,2] 3 [3,4,2] 3 0'0 2019-12-14T15:50:41.074553+0000 0'0 2019-12-21T15:48:49.587218+0000 0 2019-12-21T15:52:31.760 INFO:tasks.workunit.client.0.smithi172.stdout:2.5 73 0 0 0 0 598016 0 0 73 73 active+clean 2019-12-21T15:48:51.922004+0000 46'73 48:94 [3,4,0] 3 [3,4,0] 3 0'0 2019-12-14T15:50:41.430902+0000 0'0 2019-12-21T15:48:51.001996+0000 0 2019-12-21T15:52:31.760 INFO:tasks.workunit.client.0.smithi172.stdout:1.1 67 0 0 0 0 548864 0 0 67 67 active+clean 2019-12-21T15:48:50.513172+0000 46'67 48:87 [5,0,4] 5 [5,0,4] 5 0'0 2019-12-14T15:50:41.787088+0000 0'0 2019-12-21T15:48:49.587218+0000 0 2019-12-21T15:52:31.760 INFO:tasks.workunit.client.0.smithi172.stdout:2.2 55 0 0 0 0 450560 0 0 55 55 active+clean 2019-12-21T15:48:51.882192+0000 46'55 48:71 [5,1,0] 5 [5,1,0] 5 0'0 2019-12-14T15:50:42.145860+0000 0'0 2019-12-21T15:48:51.001996+0000 0 2019-12-21T15:52:31.760 INFO:tasks.workunit.client.0.smithi172.stdout:3.3 54 0 0 0 0 442368 0 0 54 54 active+clean 2019-12-21T15:48:53.573324+0000 46'54 48:69 [4,0,5] 4 [4,0,5] 4 0'0 2019-12-14T15:50:42.500954+0000 0'0 2019-12-21T15:48:52.433608+0000 0 2019-12-21T15:52:31.760 INFO:tasks.workunit.client.0.smithi172.stdout:1.0 72 0 0 0 0 589824 0 0 72 72 active+clean 2019-12-21T15:48:50.266436+0000 46'72 48:95 [1,0,2] 1 [1,0,2] 1 0'0 2019-12-14T15:50:42.876490+0000 0'0 2019-12-21T15:48:49.587218+0000 0 2019-12-21T15:52:31.760 INFO:tasks.workunit.client.0.smithi172.stdout:2.3 54 0 0 0 0 442368 0 0 54 54 active+clean 2019-12-21T15:48:51.921328+0000 46'54 48:70 [5,4,1] 5 [5,4,1] 5 0'0 2019-12-14T15:50:43.233249+0000 0'0 2019-12-21T15:48:51.001996+0000 0 2019-12-21T15:52:31.761 INFO:tasks.workunit.client.0.smithi172.stdout:3.2 55 0 0 0 0 450560 0 0 55 55 active+clean 2019-12-21T15:48:53.597372+0000 46'55 48:71 [3,5,0] 3 [3,5,0] 3 0'0 2019-12-14T15:50:43.589419+0000 0'0 2019-12-21T15:48:52.433608+0000 0 2019-12-21T15:52:31.761 INFO:tasks.workunit.client.0.smithi172.stdout:1.2 55 0 0 0 0 450560 0 0 55 55 active+clean 2019-12-21T15:48:50.266115+0000 46'55 48:78 [0,4,1] 0 [0,4,1] 0 0'0 2019-12-14T15:50:43.946014+0000 0'0 2019-12-21T15:48:49.587218+0000 0 2019-12-21T15:52:31.761 INFO:tasks.workunit.client.0.smithi172.stdout:2.1 67 0 0 0 0 548864 0 0 67 67 active+clean 2019-12-21T15:48:51.921897+0000 46'67 48:85 [2,5,0] 2 [2,5,0] 2 0'0 2019-12-14T15:50:44.302575+0000 0'0 2019-12-21T15:48:51.001996+0000 0 2019-12-21T15:52:31.761 INFO:tasks.workunit.client.0.smithi172.stdout:3.0 72 0 0 0 0 589824 0 0 72 72 active+clean 2019-12-21T15:48:53.598105+0000 46'72 48:87 [1,2,4] 1 [1,2,4] 1 0'0 2019-12-14T15:50:44.658995+0000 0'0 2019-12-21T15:48:52.433608+0000 0 2019-12-21T15:52:31.761 INFO:tasks.workunit.client.0.smithi172.stdout:1.3 54 0 0 0 0 442368 0 0 54 54 active+clean 2019-12-21T15:48:50.232614+0000 46'54 48:75 [4,1,2] 4 [4,1,2] 4 0'0 2019-12-14T15:50:45.014718+0000 0'0 2019-12-21T15:48:49.587218+0000 0 2019-12-21T15:52:31.761 INFO:tasks.workunit.client.0.smithi172.stdout:2.0 72 0 0 0 0 589824 0 0 72 72 active+clean 2019-12-21T15:48:51.921796+0000 46'72 48:93 [3,5,1] 3 [3,5,1] 3 0'0 2019-12-14T15:50:45.372506+0000 0'0 2019-12-21T15:48:51.001996+0000 0 2019-12-21T15:52:31.761 INFO:tasks.workunit.client.0.smithi172.stdout:3.1 67 0 0 0 0 548864 0 0 67 67 active+clean 2019-12-21T15:48:53.483350+0000 46'67 48:82 [4,3,5] 4 [4,3,5] 4 0'0 2019-12-14T15:50:45.728301+0000 0'0 2019-12-21T15:48:52.433608+0000 0 2019-12-21T15:52:31.762 INFO:tasks.workunit.client.0.smithi172.stdout:3.6 64 0 0 0 0 524288 0 0 64 64 active+clean 2019-12-21T15:48:53.596770+0000 46'64 48:79 [0,1,4] 0 [0,1,4] 0 0'0 2019-12-14T15:50:46.086797+0000 0'0 2019-12-21T15:48:52.433608+0000 0 2019-12-21T15:52:31.762 INFO:tasks.workunit.client.0.smithi172.stdout:1.4 51 0 0 0 0 417792 0 0 51 51 active+clean 2019-12-21T15:48:50.267005+0000 46'51 48:74 [3,0,1] 3 [3,0,1] 3 0'0 2019-12-14T15:50:46.443675+0000 0'0 2019-12-21T15:48:49.587218+0000 0 2019-12-21T15:52:31.762 INFO:tasks.workunit.client.0.smithi172.stdout:2.7 53 0 0 0 0 434176 0 0 53 53 active+clean 2019-12-21T15:48:51.882809+0000 46'53 48:74 [3,0,2] 3 [3,0,2] 3 0'0 2019-12-14T15:50:46.801167+0000 0'0 2019-12-21T15:48:51.001996+0000 0 2019-12-21T15:52:31.762 INFO:tasks.workunit.client.0.smithi172.stdout:3.7 53 0 0 0 0 434176 0 0 53 53 active+clean 2019-12-21T15:48:53.573718+0000 46'53 48:68 [4,3,0] 4 [4,3,0] 4 0'0 2019-12-14T15:50:47.156376+0000 0'0 2019-12-21T15:48:52.433608+0000 0 2019-12-21T15:52:31.762 INFO:tasks.workunit.client.0.smithi172.stdout:1.5 73 0 0 0 0 598016 0 0 73 73 active+clean 2019-12-21T15:48:50.232808+0000 46'73 47:93 [4,2,5] 4 [4,2,5] 4 0'0 2019-12-21T15:48:49.587218+0000 0'0 2019-12-21T15:48:49.587218+0000 0 2019-12-21T15:52:31.763 INFO:tasks.workunit.client.0.smithi172.stdout:2.6 64 0 0 0 0 524288 0 0 64 64 active+clean 2019-12-21T15:48:51.921603+0000 46'64 47:82 [1,0,4] 1 [1,0,4] 1 0'0 2019-12-21T15:48:51.001996+0000 0'0 2019-12-21T15:48:51.001996+0000 0 2019-12-21T15:52:31.763 INFO:tasks.workunit.client.0.smithi172.stdout: 2019-12-21T15:52:31.763 INFO:tasks.workunit.client.0.smithi172.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilisation. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2019-12-21T15:52:31.763 INFO:tasks.workunit.client.0.smithi172.stderr:dumped pgs 2019-12-21T15:52:31.765 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:106: TEST_recover_unexpected: grep scrubbing 2019-12-21T15:52:31.765 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:106: TEST_recover_unexpected: ceph pg dump pgs 2019-12-21T15:52:31.987 INFO:tasks.workunit.client.0.smithi172.stderr:dumped pgs 2019-12-21T15:52:31.995 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:110: TEST_recover_unexpected: sleep 1 2019-12-21T15:52:32.997 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:103: TEST_recover_unexpected: for i in $(seq 0 300) 2019-12-21T15:52:32.997 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:105: TEST_recover_unexpected: ceph pg dump pgs 2019-12-21T15:52:33.227 INFO:tasks.workunit.client.0.smithi172.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2019-12-21T15:52:33.227 INFO:tasks.workunit.client.0.smithi172.stdout:3.d 67 0 0 0 0 548864 0 0 67 67 active+clean 2019-12-21T15:48:53.596654+0000 46'67 48:83 [3,1,0] 3 [3,1,0] 3 0'0 2019-12-14T15:50:31.047070+0000 0'0 2019-12-21T15:48:52.433608+0000 0 2019-12-21T15:52:33.227 INFO:tasks.workunit.client.0.smithi172.stdout:1.f 62 0 0 0 0 507904 0 0 62 62 active+clean 2019-12-21T15:48:50.267215+0000 46'62 48:85 [3,0,4] 3 [3,0,4] 3 0'0 2019-12-14T15:50:31.406387+0000 0'0 2019-12-21T15:48:49.587218+0000 0 2019-12-21T15:52:33.227 INFO:tasks.workunit.client.0.smithi172.stdout:2.c 60 0 0 0 0 491520 0 0 60 60 active+clean 2019-12-21T15:48:51.921595+0000 46'60 48:78 [2,0,5] 2 [2,0,5] 2 0'0 2019-12-14T15:50:31.765715+0000 0'0 2019-12-21T15:48:51.001996+0000 0 2019-12-21T15:52:33.227 INFO:tasks.workunit.client.0.smithi172.stdout:3.c 60 0 0 0 0 491520 0 0 60 60 active+clean 2019-12-21T15:48:53.573137+0000 46'60 48:73 [5,3,1] 5 [5,3,1] 5 0'0 2019-12-14T15:50:32.121892+0000 0'0 2019-12-21T15:48:52.433608+0000 0 2019-12-21T15:52:33.228 INFO:tasks.workunit.client.0.smithi172.stdout:1.e 69 0 0 0 0 565248 0 0 69 69 active+clean 2019-12-21T15:48:50.233389+0000 46'69 48:93 [1,4,2] 1 [1,4,2] 1 0'0 2019-12-14T15:50:32.477832+0000 0'0 2019-12-21T15:48:49.587218+0000 0 2019-12-21T15:52:33.228 INFO:tasks.workunit.client.0.smithi172.stdout:2.d 67 0 0 0 0 548864 0 0 67 67 active+clean 2019-12-21T15:48:51.831821+0000 46'67 48:83 [5,1,3] 5 [5,1,3] 5 0'0 2019-12-14T15:50:32.835272+0000 0'0 2019-12-21T15:48:51.001996+0000 0 2019-12-21T15:52:33.228 INFO:tasks.workunit.client.0.smithi172.stdout:3.f 62 0 0 0 0 507904 0 0 62 62 active+clean 2019-12-21T15:48:53.597626+0000 46'62 48:77 [0,1,4] 0 [0,1,4] 0 0'0 2019-12-14T15:50:33.192542+0000 0'0 2019-12-21T15:48:52.433608+0000 0 2019-12-21T15:52:33.228 INFO:tasks.workunit.client.0.smithi172.stdout:1.d 67 0 0 0 0 548864 0 0 67 67 active+clean 2019-12-21T15:48:50.232330+0000 46'67 48:88 [2,4,1] 2 [2,4,1] 2 0'0 2019-12-14T15:50:33.550430+0000 0'0 2019-12-21T15:48:49.587218+0000 0 2019-12-21T15:52:33.228 INFO:tasks.workunit.client.0.smithi172.stdout:2.e 69 0 0 0 0 565248 0 0 69 69 active+clean 2019-12-21T15:48:51.921425+0000 46'69 48:87 [2,3,5] 2 [2,3,5] 2 0'0 2019-12-14T15:50:33.906226+0000 0'0 2019-12-21T15:48:51.001996+0000 0 2019-12-21T15:52:33.228 INFO:tasks.workunit.client.0.smithi172.stdout:3.e 69 0 0 0 0 565248 0 0 69 69 active+clean 2019-12-21T15:48:53.597334+0000 46'69 48:83 [2,4,1] 2 [2,4,1] 2 0'0 2019-12-14T15:50:34.263496+0000 0'0 2019-12-21T15:48:52.433608+0000 0 2019-12-21T15:52:33.228 INFO:tasks.workunit.client.0.smithi172.stdout:1.c 60 0 0 0 0 491520 0 0 60 60 active+clean 2019-12-21T15:48:50.266642+0000 46'60 48:83 [3,0,4] 3 [3,0,4] 3 0'0 2019-12-14T15:50:34.647688+0000 0'0 2019-12-21T15:48:49.587218+0000 0 2019-12-21T15:52:33.229 INFO:tasks.workunit.client.0.smithi172.stdout:2.f 62 0 0 0 0 507904 0 0 62 62 active+clean 2019-12-21T15:48:51.780390+0000 46'62 48:81 [0,5,4] 0 [0,5,4] 0 0'0 2019-12-14T15:50:35.003467+0000 0'0 2019-12-21T15:48:51.001996+0000 0 2019-12-21T15:52:33.229 INFO:tasks.workunit.client.0.smithi172.stdout:3.9 61 0 0 0 0 499712 0 0 61 61 active+clean 2019-12-21T15:48:53.597754+0000 46'61 48:77 [4,2,5] 4 [4,2,5] 4 0'0 2019-12-14T15:50:35.361209+0000 0'0 2019-12-21T15:48:52.433608+0000 0 2019-12-21T15:52:33.229 INFO:tasks.workunit.client.0.smithi172.stdout:1.b 65 0 0 0 0 532480 0 0 65 65 active+clean 2019-12-21T15:48:50.266884+0000 46'65 48:89 [1,0,2] 1 [1,0,2] 1 0'0 2019-12-14T15:50:35.718811+0000 0'0 2019-12-21T15:48:49.587218+0000 0 2019-12-21T15:52:33.229 INFO:tasks.workunit.client.0.smithi172.stdout:2.8 66 0 0 0 0 540672 0 0 66 66 active+clean 2019-12-21T15:48:51.779146+0000 46'66 48:85 [0,2,1] 0 [0,2,1] 0 0'0 2019-12-14T15:50:36.077652+0000 0'0 2019-12-21T15:48:51.001996+0000 0 2019-12-21T15:52:33.229 INFO:tasks.workunit.client.0.smithi172.stdout:3.8 66 0 0 0 0 540672 0 0 66 66 active+clean 2019-12-21T15:48:53.596865+0000 46'66 48:82 [4,1,2] 4 [4,1,2] 4 0'0 2019-12-14T15:50:36.432117+0000 0'0 2019-12-21T15:48:52.433608+0000 0 2019-12-21T15:52:33.229 INFO:tasks.workunit.client.0.smithi172.stdout:1.a 61 0 0 0 0 499712 0 0 61 61 active+clean 2019-12-21T15:48:50.232998+0000 46'61 48:85 [1,2,4] 1 [1,2,4] 1 0'0 2019-12-14T15:50:36.788931+0000 0'0 2019-12-21T15:48:49.587218+0000 0 2019-12-21T15:52:33.230 INFO:tasks.workunit.client.0.smithi172.stdout:2.9 61 0 0 0 0 499712 0 0 61 61 active+clean 2019-12-21T15:48:51.883247+0000 46'61 48:80 [4,1,0] 4 [4,1,0] 4 0'0 2019-12-14T15:50:37.147365+0000 0'0 2019-12-21T15:48:51.001996+0000 0 2019-12-21T15:52:33.230 INFO:tasks.workunit.client.0.smithi172.stdout:3.b 65 0 0 0 0 532480 0 0 65 65 active+clean 2019-12-21T15:48:53.598249+0000 46'65 48:81 [3,0,4] 3 [3,0,4] 3 0'0 2019-12-14T15:50:37.503448+0000 0'0 2019-12-21T15:48:52.433608+0000 0 2019-12-21T15:52:33.230 INFO:tasks.workunit.client.0.smithi172.stdout:1.9 61 0 0 0 0 499712 0 0 61 61 active+clean 2019-12-21T15:48:50.190683+0000 46'61 48:85 [1,4,5] 1 [1,4,5] 1 0'0 2019-12-14T15:50:37.859855+0000 0'0 2019-12-21T15:48:49.587218+0000 0 2019-12-21T15:52:33.230 INFO:tasks.workunit.client.0.smithi172.stdout:2.a 61 0 0 0 0 499712 0 0 61 61 active+clean 2019-12-21T15:48:51.921320+0000 46'61 48:81 [1,5,0] 1 [1,5,0] 1 0'0 2019-12-14T15:50:38.219021+0000 0'0 2019-12-21T15:48:51.001996+0000 0 2019-12-21T15:52:33.230 INFO:tasks.workunit.client.0.smithi172.stdout:3.a 61 0 0 0 0 499712 0 0 61 61 active+clean 2019-12-21T15:48:53.573078+0000 46'61 48:77 [3,1,5] 3 [3,1,5] 3 0'0 2019-12-14T15:50:38.576046+0000 0'0 2019-12-21T15:48:52.433608+0000 0 2019-12-21T15:52:33.230 INFO:tasks.workunit.client.0.smithi172.stdout:1.8 66 0 0 0 0 540672 0 0 66 66 active+clean 2019-12-21T15:48:51.230728+0000 46'66 48:86 [5,4,3] 5 [5,4,3] 5 0'0 2019-12-14T15:50:38.931060+0000 0'0 2019-12-21T15:48:49.587218+0000 0 2019-12-21T15:52:33.230 INFO:tasks.workunit.client.0.smithi172.stdout:2.b 65 0 0 0 0 532480 0 0 65 65 active+clean 2019-12-21T15:48:51.922814+0000 46'65 48:86 [3,2,5] 3 [3,2,5] 3 0'0 2019-12-14T15:50:39.286910+0000 0'0 2019-12-21T15:48:51.001996+0000 0 2019-12-21T15:52:33.231 INFO:tasks.workunit.client.0.smithi172.stdout:3.5 73 0 0 0 0 598016 0 0 73 73 active+clean 2019-12-21T15:48:53.483945+0000 46'73 48:86 [5,3,4] 5 [5,3,4] 5 0'0 2019-12-14T15:50:39.644099+0000 0'0 2019-12-21T15:48:52.433608+0000 0 2019-12-21T15:52:33.231 INFO:tasks.workunit.client.0.smithi172.stdout:1.7 53 0 0 0 0 434176 0 0 53 53 active+clean 2019-12-21T15:48:50.267398+0000 46'53 48:77 [1,3,0] 1 [1,3,0] 1 0'0 2019-12-14T15:50:40.000825+0000 0'0 2019-12-21T15:48:49.587218+0000 0 2019-12-21T15:52:33.231 INFO:tasks.workunit.client.0.smithi172.stdout:2.4 51 0 0 0 0 417792 0 0 51 51 active+clean 2019-12-21T15:48:51.883549+0000 46'51 48:71 [1,0,2] 1 [1,0,2] 1 0'0 2019-12-14T15:50:40.359130+0000 0'0 2019-12-21T15:48:51.001996+0000 0 2019-12-21T15:52:33.231 INFO:tasks.workunit.client.0.smithi172.stdout:3.4 51 0 0 0 0 417792 0 0 51 51 active+clean 2019-12-21T15:48:53.596868+0000 46'51 48:67 [1,2,4] 1 [1,2,4] 1 0'0 2019-12-14T15:50:40.716191+0000 0'0 2019-12-21T15:48:52.433608+0000 0 2019-12-21T15:52:33.231 INFO:tasks.workunit.client.0.smithi172.stdout:1.6 64 0 0 0 0 524288 0 0 64 64 active+clean 2019-12-21T15:48:50.232532+0000 46'64 48:87 [3,4,2] 3 [3,4,2] 3 0'0 2019-12-14T15:50:41.074553+0000 0'0 2019-12-21T15:48:49.587218+0000 0 2019-12-21T15:52:33.231 INFO:tasks.workunit.client.0.smithi172.stdout:2.5 73 0 0 0 0 598016 0 0 73 73 active+clean 2019-12-21T15:48:51.922004+0000 46'73 48:94 [3,4,0] 3 [3,4,0] 3 0'0 2019-12-14T15:50:41.430902+0000 0'0 2019-12-21T15:48:51.001996+0000 0 2019-12-21T15:52:33.231 INFO:tasks.workunit.client.0.smithi172.stdout:1.1 67 0 0 0 0 548864 0 0 67 67 active+clean 2019-12-21T15:48:50.513172+0000 46'67 48:87 [5,0,4] 5 [5,0,4] 5 0'0 2019-12-14T15:50:41.787088+0000 0'0 2019-12-21T15:48:49.587218+0000 0 2019-12-21T15:52:33.232 INFO:tasks.workunit.client.0.smithi172.stdout:2.2 55 0 0 0 0 450560 0 0 55 55 active+clean 2019-12-21T15:48:51.882192+0000 46'55 48:71 [5,1,0] 5 [5,1,0] 5 0'0 2019-12-14T15:50:42.145860+0000 0'0 2019-12-21T15:48:51.001996+0000 0 2019-12-21T15:52:33.232 INFO:tasks.workunit.client.0.smithi172.stdout:3.3 54 0 0 0 0 442368 0 0 54 54 active+clean 2019-12-21T15:48:53.573324+0000 46'54 48:70 [4,0,5] 4 [4,0,5] 4 0'0 2019-12-14T15:50:42.500954+0000 0'0 2019-12-21T15:48:52.433608+0000 0 2019-12-21T15:52:33.232 INFO:tasks.workunit.client.0.smithi172.stdout:1.0 72 0 0 0 0 589824 0 0 72 72 active+clean 2019-12-21T15:48:50.266436+0000 46'72 48:96 [1,0,2] 1 [1,0,2] 1 0'0 2019-12-14T15:50:42.876490+0000 0'0 2019-12-21T15:48:49.587218+0000 0 2019-12-21T15:52:33.232 INFO:tasks.workunit.client.0.smithi172.stdout:2.3 54 0 0 0 0 442368 0 0 54 54 active+clean 2019-12-21T15:48:51.921328+0000 46'54 48:70 [5,4,1] 5 [5,4,1] 5 0'0 2019-12-14T15:50:43.233249+0000 0'0 2019-12-21T15:48:51.001996+0000 0 2019-12-21T15:52:33.232 INFO:tasks.workunit.client.0.smithi172.stdout:3.2 55 0 0 0 0 450560 0 0 55 55 active+clean 2019-12-21T15:48:53.597372+0000 46'55 48:71 [3,5,0] 3 [3,5,0] 3 0'0 2019-12-14T15:50:43.589419+0000 0'0 2019-12-21T15:48:52.433608+0000 0 2019-12-21T15:52:33.232 INFO:tasks.workunit.client.0.smithi172.stdout:1.2 55 0 0 0 0 450560 0 0 55 55 active+clean 2019-12-21T15:48:50.266115+0000 46'55 48:78 [0,4,1] 0 [0,4,1] 0 0'0 2019-12-14T15:50:43.946014+0000 0'0 2019-12-21T15:48:49.587218+0000 0 2019-12-21T15:52:33.232 INFO:tasks.workunit.client.0.smithi172.stdout:2.1 67 0 0 0 0 548864 0 0 67 67 active+clean 2019-12-21T15:48:51.921897+0000 46'67 48:85 [2,5,0] 2 [2,5,0] 2 0'0 2019-12-14T15:50:44.302575+0000 0'0 2019-12-21T15:48:51.001996+0000 0 2019-12-21T15:52:33.233 INFO:tasks.workunit.client.0.smithi172.stdout:3.0 72 0 0 0 0 589824 0 0 72 72 active+clean 2019-12-21T15:48:53.598105+0000 46'72 48:88 [1,2,4] 1 [1,2,4] 1 0'0 2019-12-14T15:50:44.658995+0000 0'0 2019-12-21T15:48:52.433608+0000 0 2019-12-21T15:52:33.233 INFO:tasks.workunit.client.0.smithi172.stdout:1.3 54 0 0 0 0 442368 0 0 54 54 active+clean 2019-12-21T15:48:50.232614+0000 46'54 48:76 [4,1,2] 4 [4,1,2] 4 0'0 2019-12-14T15:50:45.014718+0000 0'0 2019-12-21T15:48:49.587218+0000 0 2019-12-21T15:52:33.233 INFO:tasks.workunit.client.0.smithi172.stdout:2.0 72 0 0 0 0 589824 0 0 72 72 active+clean 2019-12-21T15:48:51.921796+0000 46'72 48:93 [3,5,1] 3 [3,5,1] 3 0'0 2019-12-14T15:50:45.372506+0000 0'0 2019-12-21T15:48:51.001996+0000 0 2019-12-21T15:52:33.233 INFO:tasks.workunit.client.0.smithi172.stdout:3.1 67 0 0 0 0 548864 0 0 67 67 active+clean 2019-12-21T15:48:53.483350+0000 46'67 48:83 [4,3,5] 4 [4,3,5] 4 0'0 2019-12-14T15:50:45.728301+0000 0'0 2019-12-21T15:48:52.433608+0000 0 2019-12-21T15:52:33.233 INFO:tasks.workunit.client.0.smithi172.stdout:3.6 64 0 0 0 0 524288 0 0 64 64 active+clean 2019-12-21T15:48:53.596770+0000 46'64 48:79 [0,1,4] 0 [0,1,4] 0 0'0 2019-12-14T15:50:46.086797+0000 0'0 2019-12-21T15:48:52.433608+0000 0 2019-12-21T15:52:33.233 INFO:tasks.workunit.client.0.smithi172.stdout:1.4 51 0 0 0 0 417792 0 0 51 51 active+clean 2019-12-21T15:48:50.267005+0000 46'51 48:74 [3,0,1] 3 [3,0,1] 3 0'0 2019-12-14T15:50:46.443675+0000 0'0 2019-12-21T15:48:49.587218+0000 0 2019-12-21T15:52:33.233 INFO:tasks.workunit.client.0.smithi172.stdout:2.7 53 0 0 0 0 434176 0 0 53 53 active+clean 2019-12-21T15:48:51.882809+0000 46'53 48:74 [3,0,2] 3 [3,0,2] 3 0'0 2019-12-14T15:50:46.801167+0000 0'0 2019-12-21T15:48:51.001996+0000 0 2019-12-21T15:52:33.234 INFO:tasks.workunit.client.0.smithi172.stdout:3.7 53 0 0 0 0 434176 0 0 53 53 active+clean 2019-12-21T15:48:53.573718+0000 46'53 48:69 [4,3,0] 4 [4,3,0] 4 0'0 2019-12-14T15:50:47.156376+0000 0'0 2019-12-21T15:48:52.433608+0000 0 2019-12-21T15:52:33.234 INFO:tasks.workunit.client.0.smithi172.stdout:1.5 73 0 0 0 0 598016 0 0 73 73 active+clean 2019-12-21T15:48:50.232808+0000 46'73 48:95 [4,2,5] 4 [4,2,5] 4 0'0 2019-12-14T15:50:47.510576+0000 0'0 2019-12-21T15:48:49.587218+0000 0 2019-12-21T15:52:33.234 INFO:tasks.workunit.client.0.smithi172.stdout:2.6 64 0 0 0 0 524288 0 0 64 64 active+clean 2019-12-21T15:48:51.921603+0000 46'64 48:84 [1,0,4] 1 [1,0,4] 1 0'0 2019-12-14T15:50:47.869116+0000 0'0 2019-12-21T15:48:51.001996+0000 0 2019-12-21T15:52:33.234 INFO:tasks.workunit.client.0.smithi172.stdout: 2019-12-21T15:52:33.234 INFO:tasks.workunit.client.0.smithi172.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilisation. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2019-12-21T15:52:33.235 INFO:tasks.workunit.client.0.smithi172.stderr:dumped pgs 2019-12-21T15:52:33.236 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:106: TEST_recover_unexpected: ceph pg dump pgs 2019-12-21T15:52:33.236 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:106: TEST_recover_unexpected: grep scrubbing 2019-12-21T15:52:33.457 INFO:tasks.workunit.client.0.smithi172.stderr:dumped pgs 2019-12-21T15:52:33.465 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:110: TEST_recover_unexpected: sleep 1 2019-12-21T15:52:34.466 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:103: TEST_recover_unexpected: for i in $(seq 0 300) 2019-12-21T15:52:34.467 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:105: TEST_recover_unexpected: ceph pg dump pgs 2019-12-21T15:52:34.700 INFO:tasks.workunit.client.0.smithi172.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2019-12-21T15:52:34.700 INFO:tasks.workunit.client.0.smithi172.stdout:3.d 67 0 0 0 0 548864 0 0 67 67 active+clean 2019-12-21T15:48:53.596654+0000 46'67 48:83 [3,1,0] 3 [3,1,0] 3 0'0 2019-12-14T15:50:31.047070+0000 0'0 2019-12-21T15:48:52.433608+0000 0 2019-12-21T15:52:34.701 INFO:tasks.workunit.client.0.smithi172.stdout:1.f 62 0 0 0 0 507904 0 0 62 62 active+clean 2019-12-21T15:48:50.267215+0000 46'62 48:85 [3,0,4] 3 [3,0,4] 3 0'0 2019-12-14T15:50:31.406387+0000 0'0 2019-12-21T15:48:49.587218+0000 0 2019-12-21T15:52:34.701 INFO:tasks.workunit.client.0.smithi172.stdout:2.c 60 0 0 0 0 491520 0 0 60 60 active+clean 2019-12-21T15:48:51.921595+0000 46'60 48:78 [2,0,5] 2 [2,0,5] 2 0'0 2019-12-14T15:50:31.765715+0000 0'0 2019-12-21T15:48:51.001996+0000 0 2019-12-21T15:52:34.701 INFO:tasks.workunit.client.0.smithi172.stdout:3.c 60 0 0 0 0 491520 0 0 60 60 active+clean 2019-12-21T15:48:53.573137+0000 46'60 48:74 [5,3,1] 5 [5,3,1] 5 0'0 2019-12-14T15:50:32.121892+0000 0'0 2019-12-21T15:48:52.433608+0000 0 2019-12-21T15:52:34.701 INFO:tasks.workunit.client.0.smithi172.stdout:1.e 69 0 0 0 0 565248 0 0 69 69 active+clean 2019-12-21T15:48:50.233389+0000 46'69 48:93 [1,4,2] 1 [1,4,2] 1 0'0 2019-12-14T15:50:32.477832+0000 0'0 2019-12-21T15:48:49.587218+0000 0 2019-12-21T15:52:34.701 INFO:tasks.workunit.client.0.smithi172.stdout:2.d 67 0 0 0 0 548864 0 0 67 67 active+clean 2019-12-21T15:48:51.831821+0000 46'67 48:84 [5,1,3] 5 [5,1,3] 5 0'0 2019-12-14T15:50:32.835272+0000 0'0 2019-12-21T15:48:51.001996+0000 0 2019-12-21T15:52:34.702 INFO:tasks.workunit.client.0.smithi172.stdout:3.f 62 0 0 0 0 507904 0 0 62 62 active+clean 2019-12-21T15:48:53.597626+0000 46'62 48:77 [0,1,4] 0 [0,1,4] 0 0'0 2019-12-14T15:50:33.192542+0000 0'0 2019-12-21T15:48:52.433608+0000 0 2019-12-21T15:52:34.702 INFO:tasks.workunit.client.0.smithi172.stdout:1.d 67 0 0 0 0 548864 0 0 67 67 active+clean+scrubbing 2019-12-21T15:52:31.318101+0000 46'67 49:89 [2,4,1] 2 [2,4,1] 2 0'0 2019-12-14T15:50:33.550430+0000 0'0 2019-12-21T15:48:49.587218+0000 0 2019-12-21T15:52:34.702 INFO:tasks.workunit.client.0.smithi172.stdout:2.e 69 0 0 0 0 565248 0 0 69 69 active+clean+scrubbing 2019-12-21T15:52:33.322675+0000 46'69 49:88 [2,3,5] 2 [2,3,5] 2 0'0 2019-12-14T15:50:33.906226+0000 0'0 2019-12-21T15:48:51.001996+0000 0 2019-12-21T15:52:34.702 INFO:tasks.workunit.client.0.smithi172.stdout:3.e 69 0 0 0 0 565248 0 0 69 69 active+clean 2019-12-21T15:48:53.597334+0000 46'69 48:83 [2,4,1] 2 [2,4,1] 2 0'0 2019-12-14T15:50:34.263496+0000 0'0 2019-12-21T15:48:52.433608+0000 0 2019-12-21T15:52:34.703 INFO:tasks.workunit.client.0.smithi172.stdout:1.c 60 0 0 0 0 491520 0 0 60 60 active+clean 2019-12-21T15:48:50.266642+0000 46'60 48:83 [3,0,4] 3 [3,0,4] 3 0'0 2019-12-14T15:50:34.647688+0000 0'0 2019-12-21T15:48:49.587218+0000 0 2019-12-21T15:52:34.703 INFO:tasks.workunit.client.0.smithi172.stdout:2.f 62 0 0 0 0 507904 0 0 62 62 active+clean 2019-12-21T15:48:51.780390+0000 46'62 48:81 [0,5,4] 0 [0,5,4] 0 0'0 2019-12-14T15:50:35.003467+0000 0'0 2019-12-21T15:48:51.001996+0000 0 2019-12-21T15:52:34.703 INFO:tasks.workunit.client.0.smithi172.stdout:3.9 61 0 0 0 0 499712 0 0 61 61 active+clean 2019-12-21T15:48:53.597754+0000 46'61 48:77 [4,2,5] 4 [4,2,5] 4 0'0 2019-12-14T15:50:35.361209+0000 0'0 2019-12-21T15:48:52.433608+0000 0 2019-12-21T15:52:34.703 INFO:tasks.workunit.client.0.smithi172.stdout:1.b 65 0 0 0 0 532480 0 0 65 65 active+clean 2019-12-21T15:48:50.266884+0000 46'65 48:89 [1,0,2] 1 [1,0,2] 1 0'0 2019-12-14T15:50:35.718811+0000 0'0 2019-12-21T15:48:49.587218+0000 0 2019-12-21T15:52:34.704 INFO:tasks.workunit.client.0.smithi172.stdout:2.8 66 0 0 0 0 540672 0 0 66 66 active+clean 2019-12-21T15:48:51.779146+0000 46'66 48:85 [0,2,1] 0 [0,2,1] 0 0'0 2019-12-14T15:50:36.077652+0000 0'0 2019-12-21T15:48:51.001996+0000 0 2019-12-21T15:52:34.704 INFO:tasks.workunit.client.0.smithi172.stdout:3.8 66 0 0 0 0 540672 0 0 66 66 active+clean 2019-12-21T15:48:53.596865+0000 46'66 48:82 [4,1,2] 4 [4,1,2] 4 0'0 2019-12-14T15:50:36.432117+0000 0'0 2019-12-21T15:48:52.433608+0000 0 2019-12-21T15:52:34.704 INFO:tasks.workunit.client.0.smithi172.stdout:1.a 61 0 0 0 0 499712 0 0 61 61 active+clean 2019-12-21T15:48:50.232998+0000 46'61 48:85 [1,2,4] 1 [1,2,4] 1 0'0 2019-12-14T15:50:36.788931+0000 0'0 2019-12-21T15:48:49.587218+0000 0 2019-12-21T15:52:34.704 INFO:tasks.workunit.client.0.smithi172.stdout:2.9 61 0 0 0 0 499712 0 0 61 61 active+clean 2019-12-21T15:48:51.883247+0000 46'61 48:80 [4,1,0] 4 [4,1,0] 4 0'0 2019-12-14T15:50:37.147365+0000 0'0 2019-12-21T15:48:51.001996+0000 0 2019-12-21T15:52:34.704 INFO:tasks.workunit.client.0.smithi172.stdout:3.b 65 0 0 0 0 532480 0 0 65 65 active+clean 2019-12-21T15:48:53.598249+0000 46'65 48:81 [3,0,4] 3 [3,0,4] 3 0'0 2019-12-14T15:50:37.503448+0000 0'0 2019-12-21T15:48:52.433608+0000 0 2019-12-21T15:52:34.705 INFO:tasks.workunit.client.0.smithi172.stdout:1.9 61 0 0 0 0 499712 0 0 61 61 active+clean 2019-12-21T15:48:50.190683+0000 46'61 48:85 [1,4,5] 1 [1,4,5] 1 0'0 2019-12-14T15:50:37.859855+0000 0'0 2019-12-21T15:48:49.587218+0000 0 2019-12-21T15:52:34.705 INFO:tasks.workunit.client.0.smithi172.stdout:2.a 61 0 0 0 0 499712 0 0 61 61 active+clean 2019-12-21T15:48:51.921320+0000 46'61 48:81 [1,5,0] 1 [1,5,0] 1 0'0 2019-12-14T15:50:38.219021+0000 0'0 2019-12-21T15:48:51.001996+0000 0 2019-12-21T15:52:34.705 INFO:tasks.workunit.client.0.smithi172.stdout:3.a 61 0 0 0 0 499712 0 0 61 61 active+clean 2019-12-21T15:48:53.573078+0000 46'61 48:77 [3,1,5] 3 [3,1,5] 3 0'0 2019-12-14T15:50:38.576046+0000 0'0 2019-12-21T15:48:52.433608+0000 0 2019-12-21T15:52:34.705 INFO:tasks.workunit.client.0.smithi172.stdout:1.8 66 0 0 0 0 540672 0 0 66 66 active+clean 2019-12-21T15:48:51.230728+0000 46'66 48:87 [5,4,3] 5 [5,4,3] 5 0'0 2019-12-14T15:50:38.931060+0000 0'0 2019-12-21T15:48:49.587218+0000 0 2019-12-21T15:52:34.706 INFO:tasks.workunit.client.0.smithi172.stdout:2.b 65 0 0 0 0 532480 0 0 65 65 active+clean 2019-12-21T15:48:51.922814+0000 46'65 48:86 [3,2,5] 3 [3,2,5] 3 0'0 2019-12-14T15:50:39.286910+0000 0'0 2019-12-21T15:48:51.001996+0000 0 2019-12-21T15:52:34.706 INFO:tasks.workunit.client.0.smithi172.stdout:3.5 73 0 0 0 0 598016 0 0 73 73 active+clean 2019-12-21T15:48:53.483945+0000 46'73 48:87 [5,3,4] 5 [5,3,4] 5 0'0 2019-12-14T15:50:39.644099+0000 0'0 2019-12-21T15:48:52.433608+0000 0 2019-12-21T15:52:34.706 INFO:tasks.workunit.client.0.smithi172.stdout:1.7 53 0 0 0 0 434176 0 0 53 53 active+clean 2019-12-21T15:48:50.267398+0000 46'53 48:77 [1,3,0] 1 [1,3,0] 1 0'0 2019-12-14T15:50:40.000825+0000 0'0 2019-12-21T15:48:49.587218+0000 0 2019-12-21T15:52:34.706 INFO:tasks.workunit.client.0.smithi172.stdout:2.4 51 0 0 0 0 417792 0 0 51 51 active+clean 2019-12-21T15:48:51.883549+0000 46'51 48:71 [1,0,2] 1 [1,0,2] 1 0'0 2019-12-14T15:50:40.359130+0000 0'0 2019-12-21T15:48:51.001996+0000 0 2019-12-21T15:52:34.707 INFO:tasks.workunit.client.0.smithi172.stdout:3.4 51 0 0 0 0 417792 0 0 51 51 active+clean 2019-12-21T15:48:53.596868+0000 46'51 48:67 [1,2,4] 1 [1,2,4] 1 0'0 2019-12-14T15:50:40.716191+0000 0'0 2019-12-21T15:48:52.433608+0000 0 2019-12-21T15:52:34.707 INFO:tasks.workunit.client.0.smithi172.stdout:1.6 64 0 0 0 0 524288 0 0 64 64 active+clean 2019-12-21T15:48:50.232532+0000 46'64 48:87 [3,4,2] 3 [3,4,2] 3 0'0 2019-12-14T15:50:41.074553+0000 0'0 2019-12-21T15:48:49.587218+0000 0 2019-12-21T15:52:34.707 INFO:tasks.workunit.client.0.smithi172.stdout:2.5 73 0 0 0 0 598016 0 0 73 73 active+clean 2019-12-21T15:48:51.922004+0000 46'73 48:94 [3,4,0] 3 [3,4,0] 3 0'0 2019-12-14T15:50:41.430902+0000 0'0 2019-12-21T15:48:51.001996+0000 0 2019-12-21T15:52:34.707 INFO:tasks.workunit.client.0.smithi172.stdout:1.1 67 0 0 0 0 548864 0 0 67 67 active+clean 2019-12-21T15:48:50.513172+0000 46'67 48:88 [5,0,4] 5 [5,0,4] 5 0'0 2019-12-14T15:50:41.787088+0000 0'0 2019-12-21T15:48:49.587218+0000 0 2019-12-21T15:52:34.708 INFO:tasks.workunit.client.0.smithi172.stdout:2.2 55 0 0 0 0 450560 0 0 55 55 active+clean 2019-12-21T15:48:51.882192+0000 46'55 48:72 [5,1,0] 5 [5,1,0] 5 0'0 2019-12-14T15:50:42.145860+0000 0'0 2019-12-21T15:48:51.001996+0000 0 2019-12-21T15:52:34.708 INFO:tasks.workunit.client.0.smithi172.stdout:3.3 54 0 0 0 0 442368 0 0 54 54 active+clean 2019-12-21T15:48:53.573324+0000 46'54 48:70 [4,0,5] 4 [4,0,5] 4 0'0 2019-12-14T15:50:42.500954+0000 0'0 2019-12-21T15:48:52.433608+0000 0 2019-12-21T15:52:34.708 INFO:tasks.workunit.client.0.smithi172.stdout:1.0 72 0 0 0 0 589824 0 0 72 72 active+clean 2019-12-21T15:48:50.266436+0000 46'72 48:96 [1,0,2] 1 [1,0,2] 1 0'0 2019-12-14T15:50:42.876490+0000 0'0 2019-12-21T15:48:49.587218+0000 0 2019-12-21T15:52:34.708 INFO:tasks.workunit.client.0.smithi172.stdout:2.3 54 0 0 0 0 442368 0 0 54 54 active+clean+scrubbing 2019-12-21T15:52:31.690020+0000 46'54 49:72 [5,4,1] 5 [5,4,1] 5 0'0 2019-12-14T15:50:43.233249+0000 0'0 2019-12-21T15:48:51.001996+0000 0 2019-12-21T15:52:34.708 INFO:tasks.workunit.client.0.smithi172.stdout:3.2 55 0 0 0 0 450560 0 0 55 55 active+clean 2019-12-21T15:48:53.597372+0000 46'55 48:71 [3,5,0] 3 [3,5,0] 3 0'0 2019-12-14T15:50:43.589419+0000 0'0 2019-12-21T15:48:52.433608+0000 0 2019-12-21T15:52:34.709 INFO:tasks.workunit.client.0.smithi172.stdout:1.2 55 0 0 0 0 450560 0 0 55 55 active+clean 2019-12-21T15:48:50.266115+0000 46'55 48:78 [0,4,1] 0 [0,4,1] 0 0'0 2019-12-14T15:50:43.946014+0000 0'0 2019-12-21T15:48:49.587218+0000 0 2019-12-21T15:52:34.709 INFO:tasks.workunit.client.0.smithi172.stdout:2.1 67 0 0 0 0 548864 0 0 67 67 active+clean 2019-12-21T15:48:51.921897+0000 46'67 48:85 [2,5,0] 2 [2,5,0] 2 0'0 2019-12-14T15:50:44.302575+0000 0'0 2019-12-21T15:48:51.001996+0000 0 2019-12-21T15:52:34.709 INFO:tasks.workunit.client.0.smithi172.stdout:3.0 72 0 0 0 0 589824 0 0 72 72 active+clean 2019-12-21T15:48:53.598105+0000 46'72 48:88 [1,2,4] 1 [1,2,4] 1 0'0 2019-12-14T15:50:44.658995+0000 0'0 2019-12-21T15:48:52.433608+0000 0 2019-12-21T15:52:34.709 INFO:tasks.workunit.client.0.smithi172.stdout:1.3 54 0 0 0 0 442368 0 0 54 54 active+clean 2019-12-21T15:48:50.232614+0000 46'54 48:76 [4,1,2] 4 [4,1,2] 4 0'0 2019-12-14T15:50:45.014718+0000 0'0 2019-12-21T15:48:49.587218+0000 0 2019-12-21T15:52:34.710 INFO:tasks.workunit.client.0.smithi172.stdout:2.0 72 0 0 0 0 589824 0 0 72 72 active+clean 2019-12-21T15:48:51.921796+0000 46'72 48:93 [3,5,1] 3 [3,5,1] 3 0'0 2019-12-14T15:50:45.372506+0000 0'0 2019-12-21T15:48:51.001996+0000 0 2019-12-21T15:52:34.710 INFO:tasks.workunit.client.0.smithi172.stdout:3.1 67 0 0 0 0 548864 0 0 67 67 active+clean 2019-12-21T15:48:53.483350+0000 46'67 48:83 [4,3,5] 4 [4,3,5] 4 0'0 2019-12-14T15:50:45.728301+0000 0'0 2019-12-21T15:48:52.433608+0000 0 2019-12-21T15:52:34.710 INFO:tasks.workunit.client.0.smithi172.stdout:3.6 64 0 0 0 0 524288 0 0 64 64 active+clean 2019-12-21T15:48:53.596770+0000 46'64 48:79 [0,1,4] 0 [0,1,4] 0 0'0 2019-12-14T15:50:46.086797+0000 0'0 2019-12-21T15:48:52.433608+0000 0 2019-12-21T15:52:34.710 INFO:tasks.workunit.client.0.smithi172.stdout:1.4 51 0 0 0 0 417792 0 0 51 51 active+clean 2019-12-21T15:48:50.267005+0000 46'51 48:74 [3,0,1] 3 [3,0,1] 3 0'0 2019-12-14T15:50:46.443675+0000 0'0 2019-12-21T15:48:49.587218+0000 0 2019-12-21T15:52:34.711 INFO:tasks.workunit.client.0.smithi172.stdout:2.7 53 0 0 0 0 434176 0 0 53 53 active+clean 2019-12-21T15:48:51.882809+0000 46'53 48:74 [3,0,2] 3 [3,0,2] 3 0'0 2019-12-14T15:50:46.801167+0000 0'0 2019-12-21T15:48:51.001996+0000 0 2019-12-21T15:52:34.711 INFO:tasks.workunit.client.0.smithi172.stdout:3.7 53 0 0 0 0 434176 0 0 53 53 active+clean 2019-12-21T15:48:53.573718+0000 46'53 48:69 [4,3,0] 4 [4,3,0] 4 0'0 2019-12-14T15:50:47.156376+0000 0'0 2019-12-21T15:48:52.433608+0000 0 2019-12-21T15:52:34.711 INFO:tasks.workunit.client.0.smithi172.stdout:1.5 73 0 0 0 0 598016 0 0 73 73 active+clean 2019-12-21T15:48:50.232808+0000 46'73 48:95 [4,2,5] 4 [4,2,5] 4 0'0 2019-12-14T15:50:47.510576+0000 0'0 2019-12-21T15:48:49.587218+0000 0 2019-12-21T15:52:34.711 INFO:tasks.workunit.client.0.smithi172.stdout:2.6 64 0 0 0 0 524288 0 0 64 64 active+clean 2019-12-21T15:48:51.921603+0000 46'64 48:84 [1,0,4] 1 [1,0,4] 1 0'0 2019-12-14T15:50:47.869116+0000 0'0 2019-12-21T15:48:51.001996+0000 0 2019-12-21T15:52:34.711 INFO:tasks.workunit.client.0.smithi172.stdout: 2019-12-21T15:52:34.712 INFO:tasks.workunit.client.0.smithi172.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilisation. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2019-12-21T15:52:34.713 INFO:tasks.workunit.client.0.smithi172.stderr:dumped pgs 2019-12-21T15:52:34.713 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:106: TEST_recover_unexpected: ceph pg dump pgs 2019-12-21T15:52:34.713 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:106: TEST_recover_unexpected: grep scrubbing 2019-12-21T15:52:34.930 INFO:tasks.workunit.client.0.smithi172.stderr:dumped pgs 2019-12-21T15:52:34.939 INFO:tasks.workunit.client.0.smithi172.stdout:1.d 67 0 0 0 0 548864 0 0 67 67 active+clean+scrubbing 2019-12-21T15:52:31.318101+0000 46'67 49:89 [2,4,1] 2 [2,4,1] 2 0'0 2019-12-14T15:50:33.550430+0000 0'0 2019-12-21T15:48:49.587218+0000 0 2019-12-21T15:52:34.939 INFO:tasks.workunit.client.0.smithi172.stdout:2.e 69 0 0 0 0 565248 0 0 69 69 active+clean+scrubbing 2019-12-21T15:52:33.322675+0000 46'69 49:88 [2,3,5] 2 [2,3,5] 2 0'0 2019-12-14T15:50:33.906226+0000 0'0 2019-12-21T15:48:51.001996+0000 0 2019-12-21T15:52:34.939 INFO:tasks.workunit.client.0.smithi172.stdout:2.3 54 0 0 0 0 442368 0 0 54 54 active+clean+scrubbing 2019-12-21T15:52:31.690020+0000 46'54 49:72 [5,4,1] 5 [5,4,1] 5 0'0 2019-12-14T15:50:43.233249+0000 0'0 2019-12-21T15:48:51.001996+0000 0 2019-12-21T15:52:34.940 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:107: TEST_recover_unexpected: ok=true 2019-12-21T15:52:34.940 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:108: TEST_recover_unexpected: break 2019-12-21T15:52:34.941 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:112: TEST_recover_unexpected: test true = false 2019-12-21T15:52:34.941 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:117: TEST_recover_unexpected: local total=0 2019-12-21T15:52:34.941 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:118: TEST_recover_unexpected: local zerocount=0 2019-12-21T15:52:34.941 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:119: TEST_recover_unexpected: local maxzerocount=3 2019-12-21T15:52:34.942 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:120: TEST_recover_unexpected: true 2019-12-21T15:52:34.942 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:122: TEST_recover_unexpected: pass=0 2019-12-21T15:52:34.944 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: expr 6 - 1 2019-12-21T15:52:34.944 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: seq 0 5 2019-12-21T15:52:34.944 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:52:34.945 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.0 2019-12-21T15:52:34.946 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2019-12-21T15:52:34.946 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2019-12-21T15:52:34.946 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:52:34.946 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:52:34.946 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:52:34.947 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.0.asok 2019-12-21T15:52:34.947 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:52:34.947 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.0.asok dump_scrub_reservations 2019-12-21T15:52:35.030 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:52:35.030 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 1, 2019-12-21T15:52:35.031 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 0, 2019-12-21T15:52:35.031 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:52:35.031 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:52:35.039 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:52:35.039 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.0 2019-12-21T15:52:35.040 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2019-12-21T15:52:35.040 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2019-12-21T15:52:35.040 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:52:35.040 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:52:35.040 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:52:35.041 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.0.asok 2019-12-21T15:52:35.041 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:52:35.041 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.0.asok dump_scrub_reservations 2019-12-21T15:52:35.134 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=1 2019-12-21T15:52:35.134 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 1 -gt 4 ']' 2019-12-21T15:52:35.135 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 0 + 1 2019-12-21T15:52:35.136 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=1 2019-12-21T15:52:35.136 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:52:35.137 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.1 2019-12-21T15:52:35.137 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2019-12-21T15:52:35.137 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2019-12-21T15:52:35.138 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:52:35.138 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:52:35.138 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:52:35.139 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.1.asok 2019-12-21T15:52:35.139 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:52:35.139 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.1.asok dump_scrub_reservations 2019-12-21T15:52:35.226 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:52:35.227 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 0, 2019-12-21T15:52:35.227 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 4, 2019-12-21T15:52:35.227 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:52:35.227 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:52:35.239 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:52:35.239 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.1 2019-12-21T15:52:35.239 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2019-12-21T15:52:35.239 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2019-12-21T15:52:35.239 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:52:35.240 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:52:35.240 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:52:35.240 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.1.asok 2019-12-21T15:52:35.240 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:52:35.240 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.1.asok dump_scrub_reservations 2019-12-21T15:52:35.332 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=4 2019-12-21T15:52:35.333 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2019-12-21T15:52:35.333 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 1 + 4 2019-12-21T15:52:35.334 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=5 2019-12-21T15:52:35.334 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:52:35.335 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.2 2019-12-21T15:52:35.335 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2019-12-21T15:52:35.335 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2019-12-21T15:52:35.336 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:52:35.336 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:52:35.336 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:52:35.336 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.2.asok 2019-12-21T15:52:35.336 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:52:35.336 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.2.asok dump_scrub_reservations 2019-12-21T15:52:35.423 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:52:35.424 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 2, 2019-12-21T15:52:35.424 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 0, 2019-12-21T15:52:35.424 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:52:35.424 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:52:35.432 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:52:35.433 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.2 2019-12-21T15:52:35.433 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2019-12-21T15:52:35.433 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2019-12-21T15:52:35.433 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:52:35.433 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:52:35.433 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:52:35.434 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.2.asok 2019-12-21T15:52:35.434 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:52:35.434 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.2.asok dump_scrub_reservations 2019-12-21T15:52:35.530 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=2 2019-12-21T15:52:35.530 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 2 -gt 4 ']' 2019-12-21T15:52:35.530 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 5 + 2 2019-12-21T15:52:35.531 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=7 2019-12-21T15:52:35.531 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:52:35.532 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.3 2019-12-21T15:52:35.532 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2019-12-21T15:52:35.532 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2019-12-21T15:52:35.533 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:52:35.533 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:52:35.533 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:52:35.534 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.3.asok 2019-12-21T15:52:35.534 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:52:35.534 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.3.asok dump_scrub_reservations 2019-12-21T15:52:35.621 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:52:35.622 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 1, 2019-12-21T15:52:35.622 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 2, 2019-12-21T15:52:35.622 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:52:35.622 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:52:35.630 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:52:35.631 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.3 2019-12-21T15:52:35.631 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2019-12-21T15:52:35.631 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2019-12-21T15:52:35.631 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:52:35.631 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:52:35.631 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:52:35.631 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.3.asok 2019-12-21T15:52:35.632 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:52:35.632 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.3.asok dump_scrub_reservations 2019-12-21T15:52:35.727 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=3 2019-12-21T15:52:35.727 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2019-12-21T15:52:35.728 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 7 + 3 2019-12-21T15:52:35.729 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=10 2019-12-21T15:52:35.729 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:52:35.731 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.4 2019-12-21T15:52:35.731 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2019-12-21T15:52:35.731 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2019-12-21T15:52:35.731 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:52:35.732 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:52:35.732 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:52:35.732 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.4.asok 2019-12-21T15:52:35.732 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:52:35.732 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.4.asok dump_scrub_reservations 2019-12-21T15:52:35.820 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:52:35.820 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 0, 2019-12-21T15:52:35.821 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 4, 2019-12-21T15:52:35.821 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:52:35.821 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:52:35.829 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:52:35.829 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.4 2019-12-21T15:52:35.829 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2019-12-21T15:52:35.830 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2019-12-21T15:52:35.830 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:52:35.830 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:52:35.830 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:52:35.830 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.4.asok 2019-12-21T15:52:35.831 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:52:35.831 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.4.asok dump_scrub_reservations 2019-12-21T15:52:35.924 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=4 2019-12-21T15:52:35.924 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2019-12-21T15:52:35.925 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 10 + 4 2019-12-21T15:52:35.926 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=14 2019-12-21T15:52:35.926 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:52:35.927 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.5 2019-12-21T15:52:35.927 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2019-12-21T15:52:35.928 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2019-12-21T15:52:35.928 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:52:35.928 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:52:35.928 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:52:35.928 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.5.asok 2019-12-21T15:52:35.928 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:52:35.929 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.5.asok dump_scrub_reservations 2019-12-21T15:52:36.016 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:52:36.016 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 2, 2019-12-21T15:52:36.016 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 2, 2019-12-21T15:52:36.017 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:52:36.017 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:52:36.025 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:52:36.025 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.5 2019-12-21T15:52:36.025 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2019-12-21T15:52:36.026 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2019-12-21T15:52:36.026 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:52:36.026 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:52:36.026 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:52:36.026 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.5.asok 2019-12-21T15:52:36.026 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:52:36.027 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.5.asok dump_scrub_reservations 2019-12-21T15:52:36.120 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=4 2019-12-21T15:52:36.120 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2019-12-21T15:52:36.121 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 14 + 4 2019-12-21T15:52:36.122 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=18 2019-12-21T15:52:36.122 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:133: TEST_recover_unexpected: '[' 18 = 0 ']' 2019-12-21T15:52:36.122 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:136: TEST_recover_unexpected: '[' 0 -gt 3 ']' 2019-12-21T15:52:36.122 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: expr 0 + 18 2019-12-21T15:52:36.123 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: total=18 2019-12-21T15:52:36.124 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: expr 2 '*' 2 2019-12-21T15:52:36.124 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: sleep 4 2019-12-21T15:52:40.126 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:120: TEST_recover_unexpected: true 2019-12-21T15:52:40.126 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:122: TEST_recover_unexpected: pass=0 2019-12-21T15:52:40.127 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: expr 6 - 1 2019-12-21T15:52:40.129 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: seq 0 5 2019-12-21T15:52:40.130 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:52:40.131 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.0 2019-12-21T15:52:40.132 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2019-12-21T15:52:40.132 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2019-12-21T15:52:40.132 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:52:40.132 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:52:40.132 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:52:40.133 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.0.asok 2019-12-21T15:52:40.133 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:52:40.133 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.0.asok dump_scrub_reservations 2019-12-21T15:52:40.229 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:52:40.229 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 1, 2019-12-21T15:52:40.230 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 0, 2019-12-21T15:52:40.230 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:52:40.230 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:52:40.239 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:52:40.240 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.0 2019-12-21T15:52:40.240 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2019-12-21T15:52:40.240 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2019-12-21T15:52:40.240 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:52:40.241 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:52:40.241 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:52:40.241 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.0.asok 2019-12-21T15:52:40.242 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:52:40.242 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.0.asok dump_scrub_reservations 2019-12-21T15:52:40.336 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=1 2019-12-21T15:52:40.337 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 1 -gt 4 ']' 2019-12-21T15:52:40.337 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 0 + 1 2019-12-21T15:52:40.338 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=1 2019-12-21T15:52:40.338 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:52:40.340 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.1 2019-12-21T15:52:40.341 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2019-12-21T15:52:40.341 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2019-12-21T15:52:40.341 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:52:40.342 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:52:40.342 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:52:40.342 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.1.asok 2019-12-21T15:52:40.342 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:52:40.343 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.1.asok dump_scrub_reservations 2019-12-21T15:52:40.428 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:52:40.428 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 0, 2019-12-21T15:52:40.428 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 4, 2019-12-21T15:52:40.428 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:52:40.429 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:52:40.437 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:52:40.437 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.1 2019-12-21T15:52:40.437 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2019-12-21T15:52:40.438 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2019-12-21T15:52:40.438 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:52:40.438 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:52:40.438 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:52:40.439 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.1.asok 2019-12-21T15:52:40.439 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:52:40.439 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.1.asok dump_scrub_reservations 2019-12-21T15:52:40.532 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=4 2019-12-21T15:52:40.532 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2019-12-21T15:52:40.533 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 1 + 4 2019-12-21T15:52:40.533 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=5 2019-12-21T15:52:40.533 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:52:40.535 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.2 2019-12-21T15:52:40.536 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2019-12-21T15:52:40.536 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2019-12-21T15:52:40.536 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:52:40.536 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:52:40.536 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:52:40.537 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.2.asok 2019-12-21T15:52:40.537 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:52:40.537 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.2.asok dump_scrub_reservations 2019-12-21T15:52:40.624 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:52:40.624 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 2, 2019-12-21T15:52:40.624 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 0, 2019-12-21T15:52:40.624 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:52:40.625 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:52:40.633 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:52:40.634 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.2 2019-12-21T15:52:40.634 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2019-12-21T15:52:40.634 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2019-12-21T15:52:40.634 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:52:40.634 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:52:40.635 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:52:40.635 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.2.asok 2019-12-21T15:52:40.635 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:52:40.635 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.2.asok dump_scrub_reservations 2019-12-21T15:52:40.729 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=2 2019-12-21T15:52:40.729 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 2 -gt 4 ']' 2019-12-21T15:52:40.730 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 5 + 2 2019-12-21T15:52:40.730 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=7 2019-12-21T15:52:40.730 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:52:40.732 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.3 2019-12-21T15:52:40.732 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2019-12-21T15:52:40.733 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2019-12-21T15:52:40.733 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:52:40.733 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:52:40.733 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:52:40.734 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.3.asok 2019-12-21T15:52:40.734 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:52:40.734 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.3.asok dump_scrub_reservations 2019-12-21T15:52:40.820 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:52:40.820 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 1, 2019-12-21T15:52:40.821 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 2, 2019-12-21T15:52:40.821 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:52:40.821 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:52:40.829 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:52:40.830 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.3 2019-12-21T15:52:40.830 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2019-12-21T15:52:40.830 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2019-12-21T15:52:40.834 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:52:40.834 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:52:40.834 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:52:40.834 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.3.asok 2019-12-21T15:52:40.835 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:52:40.835 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.3.asok dump_scrub_reservations 2019-12-21T15:52:40.924 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=3 2019-12-21T15:52:40.925 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2019-12-21T15:52:40.925 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 7 + 3 2019-12-21T15:52:40.926 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=10 2019-12-21T15:52:40.926 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:52:40.927 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.4 2019-12-21T15:52:40.927 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2019-12-21T15:52:40.928 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2019-12-21T15:52:40.928 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:52:40.928 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:52:40.928 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:52:40.928 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.4.asok 2019-12-21T15:52:40.928 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:52:40.929 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.4.asok dump_scrub_reservations 2019-12-21T15:52:41.016 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:52:41.016 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 0, 2019-12-21T15:52:41.016 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 4, 2019-12-21T15:52:41.016 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:52:41.016 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:52:41.025 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:52:41.025 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.4 2019-12-21T15:52:41.025 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2019-12-21T15:52:41.025 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2019-12-21T15:52:41.025 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:52:41.025 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:52:41.026 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:52:41.026 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.4.asok 2019-12-21T15:52:41.026 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:52:41.026 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.4.asok dump_scrub_reservations 2019-12-21T15:52:41.120 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=4 2019-12-21T15:52:41.121 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2019-12-21T15:52:41.121 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 10 + 4 2019-12-21T15:52:41.122 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=14 2019-12-21T15:52:41.122 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:52:41.123 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.5 2019-12-21T15:52:41.123 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2019-12-21T15:52:41.123 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2019-12-21T15:52:41.124 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:52:41.124 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:52:41.124 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:52:41.124 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.5.asok 2019-12-21T15:52:41.125 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:52:41.125 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.5.asok dump_scrub_reservations 2019-12-21T15:52:41.216 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:52:41.216 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 2, 2019-12-21T15:52:41.217 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 2, 2019-12-21T15:52:41.217 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:52:41.217 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:52:41.221 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:52:41.221 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.5 2019-12-21T15:52:41.221 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2019-12-21T15:52:41.222 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2019-12-21T15:52:41.222 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:52:41.222 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:52:41.222 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:52:41.222 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.5.asok 2019-12-21T15:52:41.223 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:52:41.223 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.5.asok dump_scrub_reservations 2019-12-21T15:52:41.319 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=4 2019-12-21T15:52:41.320 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2019-12-21T15:52:41.320 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 14 + 4 2019-12-21T15:52:41.320 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=18 2019-12-21T15:52:41.320 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:133: TEST_recover_unexpected: '[' 18 = 0 ']' 2019-12-21T15:52:41.320 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:136: TEST_recover_unexpected: '[' 0 -gt 3 ']' 2019-12-21T15:52:41.321 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: expr 18 + 18 2019-12-21T15:52:41.321 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: total=36 2019-12-21T15:52:41.322 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: expr 2 '*' 2 2019-12-21T15:52:41.323 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: sleep 4 2019-12-21T15:52:45.324 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:120: TEST_recover_unexpected: true 2019-12-21T15:52:45.324 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:122: TEST_recover_unexpected: pass=0 2019-12-21T15:52:45.325 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: expr 6 - 1 2019-12-21T15:52:45.327 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: seq 0 5 2019-12-21T15:52:45.328 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:52:45.329 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.0 2019-12-21T15:52:45.329 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2019-12-21T15:52:45.330 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2019-12-21T15:52:45.330 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:52:45.330 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:52:45.330 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:52:45.330 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.0.asok 2019-12-21T15:52:45.331 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:52:45.331 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.0.asok dump_scrub_reservations 2019-12-21T15:52:45.429 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:52:45.429 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 1, 2019-12-21T15:52:45.429 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 0, 2019-12-21T15:52:45.429 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:52:45.429 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:52:45.438 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:52:45.438 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.0 2019-12-21T15:52:45.438 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2019-12-21T15:52:45.439 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2019-12-21T15:52:45.439 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:52:45.439 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:52:45.439 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:52:45.439 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.0.asok 2019-12-21T15:52:45.440 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:52:45.440 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.0.asok dump_scrub_reservations 2019-12-21T15:52:45.536 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=1 2019-12-21T15:52:45.536 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 1 -gt 4 ']' 2019-12-21T15:52:45.537 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 0 + 1 2019-12-21T15:52:45.537 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=1 2019-12-21T15:52:45.538 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:52:45.539 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.1 2019-12-21T15:52:45.539 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2019-12-21T15:52:45.539 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2019-12-21T15:52:45.539 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:52:45.539 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:52:45.540 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:52:45.540 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.1.asok 2019-12-21T15:52:45.540 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:52:45.540 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.1.asok dump_scrub_reservations 2019-12-21T15:52:45.629 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:52:45.629 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 0, 2019-12-21T15:52:45.629 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 4, 2019-12-21T15:52:45.630 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:52:45.630 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:52:45.638 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:52:45.638 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.1 2019-12-21T15:52:45.638 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2019-12-21T15:52:45.638 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2019-12-21T15:52:45.638 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:52:45.639 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:52:45.639 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:52:45.639 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.1.asok 2019-12-21T15:52:45.639 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:52:45.639 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.1.asok dump_scrub_reservations 2019-12-21T15:52:45.734 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=4 2019-12-21T15:52:45.735 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2019-12-21T15:52:45.735 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 1 + 4 2019-12-21T15:52:45.736 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=5 2019-12-21T15:52:45.736 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:52:45.738 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.2 2019-12-21T15:52:45.738 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2019-12-21T15:52:45.738 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2019-12-21T15:52:45.738 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:52:45.738 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:52:45.739 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:52:45.739 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.2.asok 2019-12-21T15:52:45.739 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:52:45.739 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.2.asok dump_scrub_reservations 2019-12-21T15:52:45.826 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:52:45.826 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 2, 2019-12-21T15:52:45.826 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 0, 2019-12-21T15:52:45.826 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:52:45.826 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:52:45.835 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:52:45.835 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.2 2019-12-21T15:52:45.835 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2019-12-21T15:52:45.835 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2019-12-21T15:52:45.835 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:52:45.836 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:52:45.836 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:52:45.836 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.2.asok 2019-12-21T15:52:45.836 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:52:45.836 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.2.asok dump_scrub_reservations 2019-12-21T15:52:45.929 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=2 2019-12-21T15:52:45.930 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 2 -gt 4 ']' 2019-12-21T15:52:45.930 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 5 + 2 2019-12-21T15:52:45.931 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=7 2019-12-21T15:52:45.931 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:52:45.932 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.3 2019-12-21T15:52:45.933 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2019-12-21T15:52:45.933 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2019-12-21T15:52:45.933 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:52:45.933 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:52:45.933 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:52:45.933 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.3.asok 2019-12-21T15:52:45.934 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:52:45.934 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.3.asok dump_scrub_reservations 2019-12-21T15:52:46.020 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:52:46.021 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 1, 2019-12-21T15:52:46.021 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 2, 2019-12-21T15:52:46.021 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:52:46.021 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:52:46.029 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:52:46.029 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.3 2019-12-21T15:52:46.029 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2019-12-21T15:52:46.030 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2019-12-21T15:52:46.030 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:52:46.030 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:52:46.030 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:52:46.030 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.3.asok 2019-12-21T15:52:46.030 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:52:46.030 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.3.asok dump_scrub_reservations 2019-12-21T15:52:46.125 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=3 2019-12-21T15:52:46.125 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2019-12-21T15:52:46.125 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 7 + 3 2019-12-21T15:52:46.126 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=10 2019-12-21T15:52:46.126 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:52:46.128 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.4 2019-12-21T15:52:46.128 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2019-12-21T15:52:46.128 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2019-12-21T15:52:46.128 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:52:46.128 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:52:46.128 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:52:46.129 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.4.asok 2019-12-21T15:52:46.129 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:52:46.129 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.4.asok dump_scrub_reservations 2019-12-21T15:52:46.216 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:52:46.216 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 0, 2019-12-21T15:52:46.216 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 4, 2019-12-21T15:52:46.216 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:52:46.216 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:52:46.225 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:52:46.225 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.4 2019-12-21T15:52:46.225 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2019-12-21T15:52:46.225 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2019-12-21T15:52:46.226 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:52:46.226 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:52:46.226 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:52:46.226 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.4.asok 2019-12-21T15:52:46.226 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:52:46.226 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.4.asok dump_scrub_reservations 2019-12-21T15:52:46.320 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=4 2019-12-21T15:52:46.320 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2019-12-21T15:52:46.321 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 10 + 4 2019-12-21T15:52:46.321 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=14 2019-12-21T15:52:46.322 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:52:46.323 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.5 2019-12-21T15:52:46.323 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2019-12-21T15:52:46.323 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2019-12-21T15:52:46.323 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:52:46.324 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:52:46.324 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:52:46.324 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.5.asok 2019-12-21T15:52:46.324 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:52:46.324 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.5.asok dump_scrub_reservations 2019-12-21T15:52:46.411 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:52:46.411 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 2, 2019-12-21T15:52:46.411 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 2, 2019-12-21T15:52:46.411 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:52:46.412 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:52:46.420 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:52:46.420 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.5 2019-12-21T15:52:46.420 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2019-12-21T15:52:46.421 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2019-12-21T15:52:46.421 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:52:46.421 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:52:46.421 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:52:46.421 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.5.asok 2019-12-21T15:52:46.421 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:52:46.421 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.5.asok dump_scrub_reservations 2019-12-21T15:52:46.515 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=4 2019-12-21T15:52:46.515 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2019-12-21T15:52:46.516 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 14 + 4 2019-12-21T15:52:46.517 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=18 2019-12-21T15:52:46.517 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:133: TEST_recover_unexpected: '[' 18 = 0 ']' 2019-12-21T15:52:46.517 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:136: TEST_recover_unexpected: '[' 0 -gt 3 ']' 2019-12-21T15:52:46.517 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: expr 36 + 18 2019-12-21T15:52:46.518 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: total=54 2019-12-21T15:52:46.519 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: expr 2 '*' 2 2019-12-21T15:52:46.519 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: sleep 4 2019-12-21T15:52:50.521 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:120: TEST_recover_unexpected: true 2019-12-21T15:52:50.521 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:122: TEST_recover_unexpected: pass=0 2019-12-21T15:52:50.522 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: expr 6 - 1 2019-12-21T15:52:50.524 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: seq 0 5 2019-12-21T15:52:50.525 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:52:50.526 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.0 2019-12-21T15:52:50.526 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2019-12-21T15:52:50.526 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2019-12-21T15:52:50.527 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:52:50.527 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:52:50.527 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:52:50.527 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.0.asok 2019-12-21T15:52:50.528 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:52:50.528 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.0.asok dump_scrub_reservations 2019-12-21T15:52:50.624 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:52:50.625 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 1, 2019-12-21T15:52:50.625 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 0, 2019-12-21T15:52:50.625 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:52:50.625 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:52:50.634 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:52:50.634 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.0 2019-12-21T15:52:50.634 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2019-12-21T15:52:50.635 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2019-12-21T15:52:50.635 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:52:50.635 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:52:50.635 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:52:50.635 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.0.asok 2019-12-21T15:52:50.635 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:52:50.635 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.0.asok dump_scrub_reservations 2019-12-21T15:52:50.731 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=1 2019-12-21T15:52:50.732 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 1 -gt 4 ']' 2019-12-21T15:52:50.732 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 0 + 1 2019-12-21T15:52:50.733 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=1 2019-12-21T15:52:50.733 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:52:50.734 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.1 2019-12-21T15:52:50.734 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2019-12-21T15:52:50.735 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2019-12-21T15:52:50.735 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:52:50.735 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:52:50.735 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:52:50.735 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.1.asok 2019-12-21T15:52:50.735 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:52:50.736 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.1.asok dump_scrub_reservations 2019-12-21T15:52:50.822 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:52:50.823 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 0, 2019-12-21T15:52:50.823 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 4, 2019-12-21T15:52:50.823 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:52:50.823 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:52:50.832 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:52:50.832 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.1 2019-12-21T15:52:50.832 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2019-12-21T15:52:50.833 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2019-12-21T15:52:50.833 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:52:50.833 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:52:50.833 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:52:50.833 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.1.asok 2019-12-21T15:52:50.833 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:52:50.834 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.1.asok dump_scrub_reservations 2019-12-21T15:52:50.928 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=4 2019-12-21T15:52:50.928 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2019-12-21T15:52:50.929 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 1 + 4 2019-12-21T15:52:50.929 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=5 2019-12-21T15:52:50.929 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:52:50.930 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.2 2019-12-21T15:52:50.931 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2019-12-21T15:52:50.931 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2019-12-21T15:52:50.931 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:52:50.931 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:52:50.932 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:52:50.932 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.2.asok 2019-12-21T15:52:50.932 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:52:50.932 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.2.asok dump_scrub_reservations 2019-12-21T15:52:51.019 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:52:51.019 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 2, 2019-12-21T15:52:51.019 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 0, 2019-12-21T15:52:51.019 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:52:51.019 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:52:51.028 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:52:51.028 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.2 2019-12-21T15:52:51.028 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2019-12-21T15:52:51.028 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2019-12-21T15:52:51.028 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:52:51.029 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:52:51.029 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:52:51.029 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.2.asok 2019-12-21T15:52:51.029 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:52:51.029 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.2.asok dump_scrub_reservations 2019-12-21T15:52:51.123 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=2 2019-12-21T15:52:51.123 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 2 -gt 4 ']' 2019-12-21T15:52:51.124 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 5 + 2 2019-12-21T15:52:51.125 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=7 2019-12-21T15:52:51.125 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:52:51.126 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.3 2019-12-21T15:52:51.126 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2019-12-21T15:52:51.126 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2019-12-21T15:52:51.127 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:52:51.127 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:52:51.127 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:52:51.127 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.3.asok 2019-12-21T15:52:51.127 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:52:51.127 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.3.asok dump_scrub_reservations 2019-12-21T15:52:51.214 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:52:51.214 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 1, 2019-12-21T15:52:51.214 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 2, 2019-12-21T15:52:51.215 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:52:51.215 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:52:51.223 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:52:51.224 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.3 2019-12-21T15:52:51.224 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2019-12-21T15:52:51.224 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2019-12-21T15:52:51.225 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:52:51.225 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:52:51.225 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:52:51.225 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.3.asok 2019-12-21T15:52:51.226 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:52:51.226 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.3.asok dump_scrub_reservations 2019-12-21T15:52:51.318 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=3 2019-12-21T15:52:51.318 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2019-12-21T15:52:51.319 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 7 + 3 2019-12-21T15:52:51.319 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=10 2019-12-21T15:52:51.319 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:52:51.321 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.4 2019-12-21T15:52:51.321 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2019-12-21T15:52:51.321 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2019-12-21T15:52:51.321 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:52:51.321 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:52:51.322 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:52:51.322 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.4.asok 2019-12-21T15:52:51.322 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:52:51.322 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.4.asok dump_scrub_reservations 2019-12-21T15:52:51.409 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:52:51.410 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 0, 2019-12-21T15:52:51.410 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 4, 2019-12-21T15:52:51.410 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:52:51.410 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:52:51.418 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:52:51.419 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.4 2019-12-21T15:52:51.419 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2019-12-21T15:52:51.419 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2019-12-21T15:52:51.419 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:52:51.419 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:52:51.419 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:52:51.420 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.4.asok 2019-12-21T15:52:51.420 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:52:51.420 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.4.asok dump_scrub_reservations 2019-12-21T15:52:51.515 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=3 2019-12-21T15:52:51.515 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2019-12-21T15:52:51.516 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 10 + 3 2019-12-21T15:52:51.517 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=13 2019-12-21T15:52:51.517 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:52:51.518 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.5 2019-12-21T15:52:51.518 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2019-12-21T15:52:51.519 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2019-12-21T15:52:51.519 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:52:51.519 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:52:51.519 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:52:51.519 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.5.asok 2019-12-21T15:52:51.519 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:52:51.520 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.5.asok dump_scrub_reservations 2019-12-21T15:52:51.607 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:52:51.607 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 2, 2019-12-21T15:52:51.607 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 2, 2019-12-21T15:52:51.608 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:52:51.608 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:52:51.616 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:52:51.616 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.5 2019-12-21T15:52:51.616 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2019-12-21T15:52:51.616 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2019-12-21T15:52:51.617 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:52:51.617 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:52:51.617 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:52:51.617 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.5.asok 2019-12-21T15:52:51.617 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:52:51.617 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.5.asok dump_scrub_reservations 2019-12-21T15:52:51.712 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=4 2019-12-21T15:52:51.712 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2019-12-21T15:52:51.713 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 13 + 4 2019-12-21T15:52:51.714 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=17 2019-12-21T15:52:51.715 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:133: TEST_recover_unexpected: '[' 17 = 0 ']' 2019-12-21T15:52:51.715 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:136: TEST_recover_unexpected: '[' 0 -gt 3 ']' 2019-12-21T15:52:51.715 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: expr 54 + 17 2019-12-21T15:52:51.716 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: total=71 2019-12-21T15:52:51.716 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: expr 2 '*' 2 2019-12-21T15:52:51.717 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: sleep 4 2019-12-21T15:52:55.718 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:120: TEST_recover_unexpected: true 2019-12-21T15:52:55.718 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:122: TEST_recover_unexpected: pass=0 2019-12-21T15:52:55.719 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: expr 6 - 1 2019-12-21T15:52:55.721 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: seq 0 5 2019-12-21T15:52:55.721 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:52:55.724 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.0 2019-12-21T15:52:55.724 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2019-12-21T15:52:55.724 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2019-12-21T15:52:55.724 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:52:55.725 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:52:55.725 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:52:55.725 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.0.asok 2019-12-21T15:52:55.725 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:52:55.725 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.0.asok dump_scrub_reservations 2019-12-21T15:52:55.819 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:52:55.820 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 0, 2019-12-21T15:52:55.820 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 1, 2019-12-21T15:52:55.820 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:52:55.820 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:52:55.828 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:52:55.829 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.0 2019-12-21T15:52:55.829 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2019-12-21T15:52:55.829 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2019-12-21T15:52:55.829 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:52:55.829 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:52:55.829 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:52:55.829 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.0.asok 2019-12-21T15:52:55.830 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:52:55.830 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.0.asok dump_scrub_reservations 2019-12-21T15:52:55.925 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=1 2019-12-21T15:52:55.925 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 1 -gt 4 ']' 2019-12-21T15:52:55.925 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 0 + 1 2019-12-21T15:52:55.926 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=1 2019-12-21T15:52:55.927 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:52:55.928 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.1 2019-12-21T15:52:55.931 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2019-12-21T15:52:55.931 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2019-12-21T15:52:55.931 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:52:55.931 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:52:55.932 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:52:55.932 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.1.asok 2019-12-21T15:52:55.932 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:52:55.933 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.1.asok dump_scrub_reservations 2019-12-21T15:52:56.016 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:52:56.016 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 0, 2019-12-21T15:52:56.016 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 2, 2019-12-21T15:52:56.016 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:52:56.016 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:52:56.025 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:52:56.027 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.1 2019-12-21T15:52:56.027 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2019-12-21T15:52:56.028 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2019-12-21T15:52:56.028 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:52:56.028 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:52:56.028 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:52:56.028 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.1.asok 2019-12-21T15:52:56.028 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:52:56.028 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.1.asok dump_scrub_reservations 2019-12-21T15:52:56.121 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=2 2019-12-21T15:52:56.121 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 2 -gt 4 ']' 2019-12-21T15:52:56.121 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 1 + 2 2019-12-21T15:52:56.122 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=3 2019-12-21T15:52:56.122 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:52:56.123 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.2 2019-12-21T15:52:56.124 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2019-12-21T15:52:56.125 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2019-12-21T15:52:56.125 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:52:56.125 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:52:56.125 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:52:56.125 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.2.asok 2019-12-21T15:52:56.125 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:52:56.125 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.2.asok dump_scrub_reservations 2019-12-21T15:52:56.217 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:52:56.217 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 4, 2019-12-21T15:52:56.217 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 0, 2019-12-21T15:52:56.217 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:52:56.217 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:52:56.225 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:52:56.226 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.2 2019-12-21T15:52:56.226 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2019-12-21T15:52:56.226 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2019-12-21T15:52:56.226 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:52:56.227 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:52:56.227 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:52:56.228 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.2.asok 2019-12-21T15:52:56.228 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:52:56.228 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.2.asok dump_scrub_reservations 2019-12-21T15:52:56.315 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=4 2019-12-21T15:52:56.315 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2019-12-21T15:52:56.316 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 3 + 4 2019-12-21T15:52:56.320 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=7 2019-12-21T15:52:56.320 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:52:56.320 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.3 2019-12-21T15:52:56.320 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2019-12-21T15:52:56.320 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2019-12-21T15:52:56.321 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:52:56.321 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:52:56.321 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:52:56.321 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.3.asok 2019-12-21T15:52:56.321 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:52:56.321 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.3.asok dump_scrub_reservations 2019-12-21T15:52:56.407 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:52:56.407 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 0, 2019-12-21T15:52:56.407 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 2, 2019-12-21T15:52:56.407 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:52:56.407 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:52:56.416 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:52:56.416 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.3 2019-12-21T15:52:56.416 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2019-12-21T15:52:56.416 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2019-12-21T15:52:56.416 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:52:56.416 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:52:56.417 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:52:56.417 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.3.asok 2019-12-21T15:52:56.417 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:52:56.417 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.3.asok dump_scrub_reservations 2019-12-21T15:52:56.517 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=2 2019-12-21T15:52:56.518 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 2 -gt 4 ']' 2019-12-21T15:52:56.518 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 7 + 2 2019-12-21T15:52:56.518 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=9 2019-12-21T15:52:56.518 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:52:56.519 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.4 2019-12-21T15:52:56.519 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2019-12-21T15:52:56.519 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2019-12-21T15:52:56.521 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:52:56.521 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:52:56.521 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:52:56.522 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.4.asok 2019-12-21T15:52:56.522 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:52:56.522 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.4.asok dump_scrub_reservations 2019-12-21T15:52:56.603 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:52:56.604 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 0, 2019-12-21T15:52:56.604 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 3, 2019-12-21T15:52:56.604 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:52:56.604 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:52:56.612 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:52:56.612 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.4 2019-12-21T15:52:56.612 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2019-12-21T15:52:56.612 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2019-12-21T15:52:56.612 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:52:56.612 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:52:56.613 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:52:56.613 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.4.asok 2019-12-21T15:52:56.613 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:52:56.613 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.4.asok dump_scrub_reservations 2019-12-21T15:52:56.708 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=3 2019-12-21T15:52:56.709 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2019-12-21T15:52:56.709 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 9 + 3 2019-12-21T15:52:56.714 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=12 2019-12-21T15:52:56.714 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:52:56.714 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.5 2019-12-21T15:52:56.715 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2019-12-21T15:52:56.715 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2019-12-21T15:52:56.715 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:52:56.715 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:52:56.715 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:52:56.715 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.5.asok 2019-12-21T15:52:56.715 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:52:56.716 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.5.asok dump_scrub_reservations 2019-12-21T15:52:56.800 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:52:56.800 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 1, 2019-12-21T15:52:56.801 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 2, 2019-12-21T15:52:56.801 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:52:56.801 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:52:56.810 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:52:56.810 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.5 2019-12-21T15:52:56.810 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2019-12-21T15:52:56.810 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2019-12-21T15:52:56.810 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:52:56.810 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:52:56.811 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:52:56.811 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.5.asok 2019-12-21T15:52:56.811 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:52:56.811 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.5.asok dump_scrub_reservations 2019-12-21T15:52:56.906 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=3 2019-12-21T15:52:56.906 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2019-12-21T15:52:56.906 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 12 + 3 2019-12-21T15:52:56.907 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=15 2019-12-21T15:52:56.908 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:133: TEST_recover_unexpected: '[' 15 = 0 ']' 2019-12-21T15:52:56.908 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:136: TEST_recover_unexpected: '[' 0 -gt 3 ']' 2019-12-21T15:52:56.908 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: expr 71 + 15 2019-12-21T15:52:56.909 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: total=86 2019-12-21T15:52:56.909 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: expr 2 '*' 2 2019-12-21T15:52:56.911 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: sleep 4 2019-12-21T15:53:00.911 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:120: TEST_recover_unexpected: true 2019-12-21T15:53:00.912 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:122: TEST_recover_unexpected: pass=0 2019-12-21T15:53:00.915 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: expr 6 - 1 2019-12-21T15:53:00.915 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: seq 0 5 2019-12-21T15:53:00.915 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:53:00.916 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.0 2019-12-21T15:53:00.917 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2019-12-21T15:53:00.917 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2019-12-21T15:53:00.917 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:53:00.917 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:53:00.917 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:53:00.917 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.0.asok 2019-12-21T15:53:00.918 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:53:00.918 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.0.asok dump_scrub_reservations 2019-12-21T15:53:01.014 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:53:01.014 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 0, 2019-12-21T15:53:01.014 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 2, 2019-12-21T15:53:01.014 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:53:01.014 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:53:01.023 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:53:01.023 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.0 2019-12-21T15:53:01.023 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2019-12-21T15:53:01.024 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2019-12-21T15:53:01.024 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:53:01.024 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:53:01.024 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:53:01.024 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.0.asok 2019-12-21T15:53:01.024 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:53:01.025 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.0.asok dump_scrub_reservations 2019-12-21T15:53:01.122 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=2 2019-12-21T15:53:01.123 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 2 -gt 4 ']' 2019-12-21T15:53:01.123 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 0 + 2 2019-12-21T15:53:01.124 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=2 2019-12-21T15:53:01.124 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:53:01.125 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.1 2019-12-21T15:53:01.125 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2019-12-21T15:53:01.125 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2019-12-21T15:53:01.125 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:53:01.126 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:53:01.126 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:53:01.126 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.1.asok 2019-12-21T15:53:01.126 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:53:01.126 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.1.asok dump_scrub_reservations 2019-12-21T15:53:01.215 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:53:01.216 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 1, 2019-12-21T15:53:01.216 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 2, 2019-12-21T15:53:01.216 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:53:01.216 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:53:01.224 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:53:01.225 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.1 2019-12-21T15:53:01.225 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2019-12-21T15:53:01.225 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2019-12-21T15:53:01.225 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:53:01.225 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:53:01.225 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:53:01.226 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.1.asok 2019-12-21T15:53:01.226 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:53:01.226 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.1.asok dump_scrub_reservations 2019-12-21T15:53:01.318 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=3 2019-12-21T15:53:01.318 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2019-12-21T15:53:01.319 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 2 + 3 2019-12-21T15:53:01.319 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=5 2019-12-21T15:53:01.320 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:53:01.321 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.2 2019-12-21T15:53:01.321 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2019-12-21T15:53:01.321 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2019-12-21T15:53:01.321 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:53:01.322 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:53:01.322 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:53:01.322 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.2.asok 2019-12-21T15:53:01.322 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:53:01.322 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.2.asok dump_scrub_reservations 2019-12-21T15:53:01.409 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:53:01.410 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 2, 2019-12-21T15:53:01.410 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 1, 2019-12-21T15:53:01.410 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:53:01.411 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:53:01.418 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:53:01.418 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.2 2019-12-21T15:53:01.418 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2019-12-21T15:53:01.419 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2019-12-21T15:53:01.419 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:53:01.419 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:53:01.419 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:53:01.419 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.2.asok 2019-12-21T15:53:01.419 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:53:01.419 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.2.asok dump_scrub_reservations 2019-12-21T15:53:01.514 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=3 2019-12-21T15:53:01.514 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2019-12-21T15:53:01.517 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 5 + 3 2019-12-21T15:53:01.517 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=8 2019-12-21T15:53:01.518 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:53:01.518 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.3 2019-12-21T15:53:01.518 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2019-12-21T15:53:01.519 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2019-12-21T15:53:01.519 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:53:01.519 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:53:01.520 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:53:01.520 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.3.asok 2019-12-21T15:53:01.520 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:53:01.520 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.3.asok dump_scrub_reservations 2019-12-21T15:53:01.604 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:53:01.605 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 0, 2019-12-21T15:53:01.605 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 2, 2019-12-21T15:53:01.605 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:53:01.605 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:53:01.614 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:53:01.615 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.3 2019-12-21T15:53:01.615 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2019-12-21T15:53:01.615 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2019-12-21T15:53:01.615 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:53:01.615 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:53:01.615 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:53:01.616 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.3.asok 2019-12-21T15:53:01.616 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:53:01.616 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.3.asok dump_scrub_reservations 2019-12-21T15:53:01.710 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=2 2019-12-21T15:53:01.710 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 2 -gt 4 ']' 2019-12-21T15:53:01.711 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 8 + 2 2019-12-21T15:53:01.711 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=10 2019-12-21T15:53:01.711 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:53:01.713 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.4 2019-12-21T15:53:01.713 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2019-12-21T15:53:01.713 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2019-12-21T15:53:01.713 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:53:01.714 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:53:01.714 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:53:01.714 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.4.asok 2019-12-21T15:53:01.714 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:53:01.714 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.4.asok dump_scrub_reservations 2019-12-21T15:53:01.803 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:53:01.804 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 1, 2019-12-21T15:53:01.804 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 3, 2019-12-21T15:53:01.804 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:53:01.804 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:53:01.815 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:53:01.815 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.4 2019-12-21T15:53:01.815 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2019-12-21T15:53:01.816 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2019-12-21T15:53:01.816 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:53:01.816 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:53:01.816 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:53:01.816 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.4.asok 2019-12-21T15:53:01.816 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:53:01.817 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.4.asok dump_scrub_reservations 2019-12-21T15:53:01.906 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=4 2019-12-21T15:53:01.906 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2019-12-21T15:53:01.907 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 10 + 4 2019-12-21T15:53:01.907 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=14 2019-12-21T15:53:01.908 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:53:01.910 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.5 2019-12-21T15:53:01.910 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2019-12-21T15:53:01.910 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2019-12-21T15:53:01.910 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:53:01.910 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:53:01.910 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:53:01.911 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.5.asok 2019-12-21T15:53:01.911 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:53:01.911 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.5.asok dump_scrub_reservations 2019-12-21T15:53:01.997 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:53:01.997 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 2, 2019-12-21T15:53:01.997 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 2, 2019-12-21T15:53:01.997 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:53:01.997 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:53:02.005 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:53:02.005 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.5 2019-12-21T15:53:02.006 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2019-12-21T15:53:02.006 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2019-12-21T15:53:02.006 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:53:02.006 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:53:02.007 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:53:02.007 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.5.asok 2019-12-21T15:53:02.007 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:53:02.007 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.5.asok dump_scrub_reservations 2019-12-21T15:53:02.101 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=4 2019-12-21T15:53:02.101 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2019-12-21T15:53:02.101 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 14 + 4 2019-12-21T15:53:02.101 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=18 2019-12-21T15:53:02.101 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:133: TEST_recover_unexpected: '[' 18 = 0 ']' 2019-12-21T15:53:02.102 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:136: TEST_recover_unexpected: '[' 0 -gt 3 ']' 2019-12-21T15:53:02.102 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: expr 86 + 18 2019-12-21T15:53:02.102 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: total=104 2019-12-21T15:53:02.103 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: expr 2 '*' 2 2019-12-21T15:53:02.104 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: sleep 4 2019-12-21T15:53:06.105 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:120: TEST_recover_unexpected: true 2019-12-21T15:53:06.105 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:122: TEST_recover_unexpected: pass=0 2019-12-21T15:53:06.106 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: expr 6 - 1 2019-12-21T15:53:06.108 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: seq 0 5 2019-12-21T15:53:06.109 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:53:06.110 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.0 2019-12-21T15:53:06.110 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2019-12-21T15:53:06.110 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2019-12-21T15:53:06.111 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:53:06.111 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:53:06.111 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:53:06.111 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.0.asok 2019-12-21T15:53:06.112 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:53:06.112 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.0.asok dump_scrub_reservations 2019-12-21T15:53:06.209 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:53:06.209 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 0, 2019-12-21T15:53:06.210 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 3, 2019-12-21T15:53:06.210 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:53:06.210 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:53:06.218 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:53:06.218 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.0 2019-12-21T15:53:06.219 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2019-12-21T15:53:06.219 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2019-12-21T15:53:06.219 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:53:06.219 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:53:06.219 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:53:06.219 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.0.asok 2019-12-21T15:53:06.220 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:53:06.220 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.0.asok dump_scrub_reservations 2019-12-21T15:53:06.314 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=3 2019-12-21T15:53:06.315 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2019-12-21T15:53:06.315 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 0 + 3 2019-12-21T15:53:06.316 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=3 2019-12-21T15:53:06.316 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:53:06.317 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.1 2019-12-21T15:53:06.317 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2019-12-21T15:53:06.317 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2019-12-21T15:53:06.318 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:53:06.318 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:53:06.318 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:53:06.318 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.1.asok 2019-12-21T15:53:06.319 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:53:06.319 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.1.asok dump_scrub_reservations 2019-12-21T15:53:06.405 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:53:06.405 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 2, 2019-12-21T15:53:06.406 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 2, 2019-12-21T15:53:06.406 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:53:06.406 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:53:06.414 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:53:06.414 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.1 2019-12-21T15:53:06.414 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2019-12-21T15:53:06.414 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2019-12-21T15:53:06.415 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:53:06.415 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:53:06.415 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:53:06.415 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.1.asok 2019-12-21T15:53:06.415 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:53:06.415 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.1.asok dump_scrub_reservations 2019-12-21T15:53:06.509 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=4 2019-12-21T15:53:06.509 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2019-12-21T15:53:06.510 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 3 + 4 2019-12-21T15:53:06.510 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=7 2019-12-21T15:53:06.511 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:53:06.512 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.2 2019-12-21T15:53:06.512 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2019-12-21T15:53:06.512 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2019-12-21T15:53:06.512 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:53:06.512 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:53:06.513 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:53:06.513 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.2.asok 2019-12-21T15:53:06.513 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:53:06.513 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.2.asok dump_scrub_reservations 2019-12-21T15:53:06.601 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:53:06.601 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 2, 2019-12-21T15:53:06.601 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 2, 2019-12-21T15:53:06.601 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:53:06.601 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:53:06.610 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:53:06.610 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.2 2019-12-21T15:53:06.610 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2019-12-21T15:53:06.610 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2019-12-21T15:53:06.610 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:53:06.611 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:53:06.611 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:53:06.611 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.2.asok 2019-12-21T15:53:06.611 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:53:06.611 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.2.asok dump_scrub_reservations 2019-12-21T15:53:06.707 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=4 2019-12-21T15:53:06.707 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2019-12-21T15:53:06.708 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 7 + 4 2019-12-21T15:53:06.709 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=11 2019-12-21T15:53:06.709 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:53:06.710 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.3 2019-12-21T15:53:06.710 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2019-12-21T15:53:06.710 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2019-12-21T15:53:06.710 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:53:06.710 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:53:06.711 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:53:06.711 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.3.asok 2019-12-21T15:53:06.711 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:53:06.711 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.3.asok dump_scrub_reservations 2019-12-21T15:53:06.798 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:53:06.799 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 2, 2019-12-21T15:53:06.799 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 2, 2019-12-21T15:53:06.800 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:53:06.800 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:53:06.807 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:53:06.807 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.3 2019-12-21T15:53:06.807 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2019-12-21T15:53:06.807 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2019-12-21T15:53:06.807 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:53:06.808 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:53:06.808 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:53:06.808 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.3.asok 2019-12-21T15:53:06.808 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:53:06.809 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.3.asok dump_scrub_reservations 2019-12-21T15:53:06.903 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=4 2019-12-21T15:53:06.903 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2019-12-21T15:53:06.904 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 11 + 4 2019-12-21T15:53:06.905 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=15 2019-12-21T15:53:06.905 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:53:06.908 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.4 2019-12-21T15:53:06.908 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2019-12-21T15:53:06.909 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2019-12-21T15:53:06.909 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:53:06.909 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:53:06.910 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:53:06.910 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.4.asok 2019-12-21T15:53:06.910 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:53:06.910 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.4.asok dump_scrub_reservations 2019-12-21T15:53:06.995 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:53:06.995 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 1, 2019-12-21T15:53:06.995 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 3, 2019-12-21T15:53:06.995 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:53:06.996 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:53:07.004 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:53:07.004 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.4 2019-12-21T15:53:07.004 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2019-12-21T15:53:07.004 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2019-12-21T15:53:07.004 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:53:07.004 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:53:07.005 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:53:07.005 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.4.asok 2019-12-21T15:53:07.005 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:53:07.005 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.4.asok dump_scrub_reservations 2019-12-21T15:53:07.100 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=4 2019-12-21T15:53:07.100 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2019-12-21T15:53:07.101 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 15 + 4 2019-12-21T15:53:07.101 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=19 2019-12-21T15:53:07.101 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:53:07.102 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.5 2019-12-21T15:53:07.102 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2019-12-21T15:53:07.103 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2019-12-21T15:53:07.103 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:53:07.103 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:53:07.103 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:53:07.103 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.5.asok 2019-12-21T15:53:07.103 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:53:07.104 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.5.asok dump_scrub_reservations 2019-12-21T15:53:07.190 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:53:07.190 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 1, 2019-12-21T15:53:07.191 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 3, 2019-12-21T15:53:07.191 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:53:07.191 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:53:07.199 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:53:07.199 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.5 2019-12-21T15:53:07.199 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2019-12-21T15:53:07.199 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2019-12-21T15:53:07.199 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:53:07.199 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:53:07.200 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:53:07.200 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.5.asok 2019-12-21T15:53:07.200 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:53:07.200 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.5.asok dump_scrub_reservations 2019-12-21T15:53:07.294 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=4 2019-12-21T15:53:07.295 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2019-12-21T15:53:07.295 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 19 + 4 2019-12-21T15:53:07.296 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=23 2019-12-21T15:53:07.296 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:133: TEST_recover_unexpected: '[' 23 = 0 ']' 2019-12-21T15:53:07.296 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:136: TEST_recover_unexpected: '[' 0 -gt 3 ']' 2019-12-21T15:53:07.297 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: expr 104 + 23 2019-12-21T15:53:07.297 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: total=127 2019-12-21T15:53:07.298 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: expr 2 '*' 2 2019-12-21T15:53:07.299 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: sleep 4 2019-12-21T15:53:11.300 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:120: TEST_recover_unexpected: true 2019-12-21T15:53:11.301 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:122: TEST_recover_unexpected: pass=0 2019-12-21T15:53:11.302 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: expr 6 - 1 2019-12-21T15:53:11.303 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: seq 0 5 2019-12-21T15:53:11.304 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:53:11.305 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.0 2019-12-21T15:53:11.305 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2019-12-21T15:53:11.305 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2019-12-21T15:53:11.306 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:53:11.306 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:53:11.306 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:53:11.307 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.0.asok 2019-12-21T15:53:11.307 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:53:11.307 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.0.asok dump_scrub_reservations 2019-12-21T15:53:11.403 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:53:11.403 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 0, 2019-12-21T15:53:11.403 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 4, 2019-12-21T15:53:11.403 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:53:11.403 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:53:11.412 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:53:11.412 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.0 2019-12-21T15:53:11.412 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2019-12-21T15:53:11.413 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2019-12-21T15:53:11.413 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:53:11.413 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:53:11.413 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:53:11.413 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.0.asok 2019-12-21T15:53:11.413 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:53:11.413 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.0.asok dump_scrub_reservations 2019-12-21T15:53:11.508 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=4 2019-12-21T15:53:11.508 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2019-12-21T15:53:11.509 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 0 + 4 2019-12-21T15:53:11.509 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=4 2019-12-21T15:53:11.509 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:53:11.511 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.1 2019-12-21T15:53:11.511 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2019-12-21T15:53:11.511 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2019-12-21T15:53:11.511 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:53:11.511 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:53:11.512 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:53:11.512 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.1.asok 2019-12-21T15:53:11.512 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:53:11.512 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.1.asok dump_scrub_reservations 2019-12-21T15:53:11.599 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:53:11.600 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 2, 2019-12-21T15:53:11.600 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 2, 2019-12-21T15:53:11.600 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:53:11.600 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:53:11.608 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:53:11.608 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.1 2019-12-21T15:53:11.608 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2019-12-21T15:53:11.608 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2019-12-21T15:53:11.609 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:53:11.609 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:53:11.609 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:53:11.609 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.1.asok 2019-12-21T15:53:11.609 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:53:11.609 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.1.asok dump_scrub_reservations 2019-12-21T15:53:11.703 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=4 2019-12-21T15:53:11.704 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2019-12-21T15:53:11.704 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 4 + 4 2019-12-21T15:53:11.705 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=8 2019-12-21T15:53:11.705 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:53:11.706 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.2 2019-12-21T15:53:11.706 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2019-12-21T15:53:11.706 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2019-12-21T15:53:11.707 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:53:11.707 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:53:11.707 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:53:11.707 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.2.asok 2019-12-21T15:53:11.708 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:53:11.708 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.2.asok dump_scrub_reservations 2019-12-21T15:53:11.797 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:53:11.797 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 2, 2019-12-21T15:53:11.797 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 2, 2019-12-21T15:53:11.797 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:53:11.797 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:53:11.808 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:53:11.808 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.2 2019-12-21T15:53:11.809 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2019-12-21T15:53:11.809 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2019-12-21T15:53:11.809 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:53:11.809 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:53:11.809 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:53:11.810 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.2.asok 2019-12-21T15:53:11.810 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:53:11.810 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.2.asok dump_scrub_reservations 2019-12-21T15:53:11.900 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=4 2019-12-21T15:53:11.901 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2019-12-21T15:53:11.901 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 8 + 4 2019-12-21T15:53:11.903 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=12 2019-12-21T15:53:11.903 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:53:11.904 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.3 2019-12-21T15:53:11.904 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2019-12-21T15:53:11.904 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2019-12-21T15:53:11.904 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:53:11.905 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:53:11.905 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:53:11.905 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.3.asok 2019-12-21T15:53:11.905 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:53:11.905 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.3.asok dump_scrub_reservations 2019-12-21T15:53:11.991 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:53:11.992 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 2, 2019-12-21T15:53:11.992 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 2, 2019-12-21T15:53:11.992 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:53:11.992 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:53:12.001 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:53:12.001 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.3 2019-12-21T15:53:12.001 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2019-12-21T15:53:12.001 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2019-12-21T15:53:12.002 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:53:12.002 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:53:12.002 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:53:12.002 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.3.asok 2019-12-21T15:53:12.002 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:53:12.002 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.3.asok dump_scrub_reservations 2019-12-21T15:53:12.096 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=4 2019-12-21T15:53:12.096 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2019-12-21T15:53:12.097 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 12 + 4 2019-12-21T15:53:12.098 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=16 2019-12-21T15:53:12.098 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:53:12.100 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.4 2019-12-21T15:53:12.100 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2019-12-21T15:53:12.100 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2019-12-21T15:53:12.101 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:53:12.101 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:53:12.101 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:53:12.102 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.4.asok 2019-12-21T15:53:12.102 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:53:12.102 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.4.asok dump_scrub_reservations 2019-12-21T15:53:12.188 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:53:12.188 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 1, 2019-12-21T15:53:12.189 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 3, 2019-12-21T15:53:12.189 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:53:12.189 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:53:12.197 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:53:12.197 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.4 2019-12-21T15:53:12.197 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2019-12-21T15:53:12.198 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2019-12-21T15:53:12.198 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:53:12.198 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:53:12.198 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:53:12.198 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.4.asok 2019-12-21T15:53:12.198 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:53:12.199 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.4.asok dump_scrub_reservations 2019-12-21T15:53:12.293 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=4 2019-12-21T15:53:12.293 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2019-12-21T15:53:12.294 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 16 + 4 2019-12-21T15:53:12.294 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=20 2019-12-21T15:53:12.295 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:53:12.296 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.5 2019-12-21T15:53:12.296 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2019-12-21T15:53:12.296 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2019-12-21T15:53:12.296 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:53:12.297 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:53:12.297 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:53:12.297 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.5.asok 2019-12-21T15:53:12.297 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:53:12.297 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.5.asok dump_scrub_reservations 2019-12-21T15:53:12.384 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:53:12.384 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 1, 2019-12-21T15:53:12.385 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 3, 2019-12-21T15:53:12.385 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:53:12.385 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:53:12.393 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:53:12.393 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.5 2019-12-21T15:53:12.394 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2019-12-21T15:53:12.394 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2019-12-21T15:53:12.394 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:53:12.394 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:53:12.394 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:53:12.394 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.5.asok 2019-12-21T15:53:12.395 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:53:12.395 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.5.asok dump_scrub_reservations 2019-12-21T15:53:12.489 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=4 2019-12-21T15:53:12.489 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2019-12-21T15:53:12.489 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 20 + 4 2019-12-21T15:53:12.490 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=24 2019-12-21T15:53:12.490 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:133: TEST_recover_unexpected: '[' 24 = 0 ']' 2019-12-21T15:53:12.491 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:136: TEST_recover_unexpected: '[' 0 -gt 3 ']' 2019-12-21T15:53:12.491 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: expr 127 + 24 2019-12-21T15:53:12.492 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: total=151 2019-12-21T15:53:12.492 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: expr 2 '*' 2 2019-12-21T15:53:12.493 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: sleep 4 2019-12-21T15:53:16.495 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:120: TEST_recover_unexpected: true 2019-12-21T15:53:16.495 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:122: TEST_recover_unexpected: pass=0 2019-12-21T15:53:16.496 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: expr 6 - 1 2019-12-21T15:53:16.497 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: seq 0 5 2019-12-21T15:53:16.498 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:53:16.500 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.0 2019-12-21T15:53:16.500 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2019-12-21T15:53:16.500 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2019-12-21T15:53:16.500 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:53:16.500 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:53:16.500 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:53:16.501 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.0.asok 2019-12-21T15:53:16.501 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:53:16.501 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.0.asok dump_scrub_reservations 2019-12-21T15:53:16.597 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:53:16.597 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 0, 2019-12-21T15:53:16.597 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 4, 2019-12-21T15:53:16.597 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:53:16.597 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:53:16.607 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:53:16.607 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.0 2019-12-21T15:53:16.607 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2019-12-21T15:53:16.607 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2019-12-21T15:53:16.607 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:53:16.607 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:53:16.608 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:53:16.608 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.0.asok 2019-12-21T15:53:16.608 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:53:16.608 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.0.asok dump_scrub_reservations 2019-12-21T15:53:16.703 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=4 2019-12-21T15:53:16.704 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2019-12-21T15:53:16.704 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 0 + 4 2019-12-21T15:53:16.705 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=4 2019-12-21T15:53:16.705 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:53:16.707 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.1 2019-12-21T15:53:16.707 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2019-12-21T15:53:16.707 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2019-12-21T15:53:16.707 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:53:16.707 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:53:16.707 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:53:16.708 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.1.asok 2019-12-21T15:53:16.708 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:53:16.708 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.1.asok dump_scrub_reservations 2019-12-21T15:53:16.795 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:53:16.795 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 2, 2019-12-21T15:53:16.795 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 2, 2019-12-21T15:53:16.795 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:53:16.795 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:53:16.803 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:53:16.804 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.1 2019-12-21T15:53:16.804 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2019-12-21T15:53:16.804 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2019-12-21T15:53:16.804 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:53:16.804 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:53:16.804 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:53:16.805 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.1.asok 2019-12-21T15:53:16.805 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:53:16.805 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.1.asok dump_scrub_reservations 2019-12-21T15:53:16.899 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=4 2019-12-21T15:53:16.899 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2019-12-21T15:53:16.900 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 4 + 4 2019-12-21T15:53:16.901 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=8 2019-12-21T15:53:16.901 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:53:16.902 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.2 2019-12-21T15:53:16.902 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2019-12-21T15:53:16.902 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2019-12-21T15:53:16.902 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:53:16.903 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:53:16.903 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:53:16.903 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.2.asok 2019-12-21T15:53:16.903 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:53:16.904 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.2.asok dump_scrub_reservations 2019-12-21T15:53:16.990 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:53:16.990 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 2, 2019-12-21T15:53:16.990 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 2, 2019-12-21T15:53:16.990 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:53:16.990 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:53:16.998 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:53:16.999 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.2 2019-12-21T15:53:16.999 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2019-12-21T15:53:16.999 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2019-12-21T15:53:16.999 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:53:16.999 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:53:17.000 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:53:17.000 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.2.asok 2019-12-21T15:53:17.000 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:53:17.000 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.2.asok dump_scrub_reservations 2019-12-21T15:53:17.095 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=4 2019-12-21T15:53:17.095 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2019-12-21T15:53:17.095 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 8 + 4 2019-12-21T15:53:17.096 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=12 2019-12-21T15:53:17.096 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:53:17.097 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.3 2019-12-21T15:53:17.097 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2019-12-21T15:53:17.097 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2019-12-21T15:53:17.098 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:53:17.098 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:53:17.098 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:53:17.098 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.3.asok 2019-12-21T15:53:17.099 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:53:17.099 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.3.asok dump_scrub_reservations 2019-12-21T15:53:17.188 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:53:17.188 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 2, 2019-12-21T15:53:17.189 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 2, 2019-12-21T15:53:17.189 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:53:17.189 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:53:17.198 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:53:17.198 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.3 2019-12-21T15:53:17.199 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2019-12-21T15:53:17.199 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2019-12-21T15:53:17.199 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:53:17.199 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:53:17.199 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:53:17.199 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.3.asok 2019-12-21T15:53:17.200 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:53:17.200 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.3.asok dump_scrub_reservations 2019-12-21T15:53:17.292 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=4 2019-12-21T15:53:17.292 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2019-12-21T15:53:17.293 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 12 + 4 2019-12-21T15:53:17.294 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=16 2019-12-21T15:53:17.294 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:53:17.295 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.4 2019-12-21T15:53:17.295 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2019-12-21T15:53:17.296 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2019-12-21T15:53:17.296 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:53:17.296 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:53:17.296 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:53:17.296 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.4.asok 2019-12-21T15:53:17.296 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:53:17.296 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.4.asok dump_scrub_reservations 2019-12-21T15:53:17.384 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:53:17.384 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 1, 2019-12-21T15:53:17.384 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 3, 2019-12-21T15:53:17.384 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:53:17.384 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:53:17.392 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:53:17.393 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.4 2019-12-21T15:53:17.393 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2019-12-21T15:53:17.393 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2019-12-21T15:53:17.393 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:53:17.393 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:53:17.393 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:53:17.394 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.4.asok 2019-12-21T15:53:17.394 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:53:17.394 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.4.asok dump_scrub_reservations 2019-12-21T15:53:17.489 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=4 2019-12-21T15:53:17.489 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2019-12-21T15:53:17.489 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 16 + 4 2019-12-21T15:53:17.490 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=20 2019-12-21T15:53:17.490 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:53:17.491 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.5 2019-12-21T15:53:17.492 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2019-12-21T15:53:17.492 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2019-12-21T15:53:17.492 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:53:17.492 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:53:17.492 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:53:17.492 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.5.asok 2019-12-21T15:53:17.493 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:53:17.493 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.5.asok dump_scrub_reservations 2019-12-21T15:53:17.580 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:53:17.581 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 1, 2019-12-21T15:53:17.581 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 3, 2019-12-21T15:53:17.581 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:53:17.581 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:53:17.589 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:53:17.589 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.5 2019-12-21T15:53:17.593 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2019-12-21T15:53:17.594 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2019-12-21T15:53:17.594 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:53:17.594 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:53:17.594 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:53:17.594 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.5.asok 2019-12-21T15:53:17.594 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:53:17.595 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.5.asok dump_scrub_reservations 2019-12-21T15:53:17.684 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=4 2019-12-21T15:53:17.684 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2019-12-21T15:53:17.685 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 20 + 4 2019-12-21T15:53:17.687 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=24 2019-12-21T15:53:17.687 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:133: TEST_recover_unexpected: '[' 24 = 0 ']' 2019-12-21T15:53:17.687 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:136: TEST_recover_unexpected: '[' 0 -gt 3 ']' 2019-12-21T15:53:17.687 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: expr 151 + 24 2019-12-21T15:53:17.688 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: total=175 2019-12-21T15:53:17.688 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: expr 2 '*' 2 2019-12-21T15:53:17.689 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: sleep 4 2019-12-21T15:53:21.690 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:120: TEST_recover_unexpected: true 2019-12-21T15:53:21.690 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:122: TEST_recover_unexpected: pass=0 2019-12-21T15:53:21.691 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: expr 6 - 1 2019-12-21T15:53:21.693 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: seq 0 5 2019-12-21T15:53:21.694 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:53:21.696 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.0 2019-12-21T15:53:21.696 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2019-12-21T15:53:21.696 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2019-12-21T15:53:21.696 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:53:21.697 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:53:21.697 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:53:21.697 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.0.asok 2019-12-21T15:53:21.697 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:53:21.697 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.0.asok dump_scrub_reservations 2019-12-21T15:53:21.793 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:53:21.794 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 0, 2019-12-21T15:53:21.794 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 4, 2019-12-21T15:53:21.794 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:53:21.794 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:53:21.803 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:53:21.803 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.0 2019-12-21T15:53:21.803 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2019-12-21T15:53:21.803 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2019-12-21T15:53:21.803 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:53:21.804 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:53:21.804 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:53:21.804 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.0.asok 2019-12-21T15:53:21.804 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:53:21.804 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.0.asok dump_scrub_reservations 2019-12-21T15:53:21.900 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=3 2019-12-21T15:53:21.900 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2019-12-21T15:53:21.901 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 0 + 3 2019-12-21T15:53:21.901 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=3 2019-12-21T15:53:21.901 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:53:21.903 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.1 2019-12-21T15:53:21.903 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2019-12-21T15:53:21.903 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2019-12-21T15:53:21.903 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:53:21.903 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:53:21.904 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:53:21.904 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.1.asok 2019-12-21T15:53:21.904 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:53:21.904 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.1.asok dump_scrub_reservations 2019-12-21T15:53:21.991 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:53:21.991 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 2, 2019-12-21T15:53:21.991 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 1, 2019-12-21T15:53:21.991 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:53:21.992 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:53:22.000 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:53:22.000 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.1 2019-12-21T15:53:22.000 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2019-12-21T15:53:22.000 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2019-12-21T15:53:22.000 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:53:22.001 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:53:22.001 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:53:22.001 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.1.asok 2019-12-21T15:53:22.001 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:53:22.001 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.1.asok dump_scrub_reservations 2019-12-21T15:53:22.097 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=3 2019-12-21T15:53:22.097 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2019-12-21T15:53:22.097 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 3 + 3 2019-12-21T15:53:22.098 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=6 2019-12-21T15:53:22.098 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:53:22.099 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.2 2019-12-21T15:53:22.099 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2019-12-21T15:53:22.099 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2019-12-21T15:53:22.099 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:53:22.099 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:53:22.100 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:53:22.100 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.2.asok 2019-12-21T15:53:22.100 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:53:22.101 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.2.asok dump_scrub_reservations 2019-12-21T15:53:22.188 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:53:22.189 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 2, 2019-12-21T15:53:22.189 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 2, 2019-12-21T15:53:22.189 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:53:22.189 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:53:22.197 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:53:22.197 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.2 2019-12-21T15:53:22.197 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2019-12-21T15:53:22.198 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2019-12-21T15:53:22.198 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:53:22.198 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:53:22.198 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:53:22.198 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.2.asok 2019-12-21T15:53:22.199 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:53:22.199 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.2.asok dump_scrub_reservations 2019-12-21T15:53:22.294 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=4 2019-12-21T15:53:22.295 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2019-12-21T15:53:22.296 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 6 + 4 2019-12-21T15:53:22.296 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=10 2019-12-21T15:53:22.296 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:53:22.298 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.3 2019-12-21T15:53:22.298 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2019-12-21T15:53:22.300 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2019-12-21T15:53:22.300 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:53:22.300 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:53:22.300 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:53:22.301 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.3.asok 2019-12-21T15:53:22.301 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:53:22.301 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.3.asok dump_scrub_reservations 2019-12-21T15:53:22.389 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:53:22.389 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 2, 2019-12-21T15:53:22.389 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 2, 2019-12-21T15:53:22.389 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:53:22.389 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:53:22.395 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:53:22.395 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.3 2019-12-21T15:53:22.395 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2019-12-21T15:53:22.395 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2019-12-21T15:53:22.395 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:53:22.395 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:53:22.396 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:53:22.396 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.3.asok 2019-12-21T15:53:22.396 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:53:22.396 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.3.asok dump_scrub_reservations 2019-12-21T15:53:22.490 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=4 2019-12-21T15:53:22.491 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2019-12-21T15:53:22.491 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 10 + 4 2019-12-21T15:53:22.492 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=14 2019-12-21T15:53:22.492 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:53:22.493 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.4 2019-12-21T15:53:22.493 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2019-12-21T15:53:22.493 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2019-12-21T15:53:22.494 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:53:22.494 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:53:22.494 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:53:22.494 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.4.asok 2019-12-21T15:53:22.494 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:53:22.494 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.4.asok dump_scrub_reservations 2019-12-21T15:53:22.582 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:53:22.582 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 1, 2019-12-21T15:53:22.582 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 3, 2019-12-21T15:53:22.582 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:53:22.583 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:53:22.591 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:53:22.591 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.4 2019-12-21T15:53:22.591 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2019-12-21T15:53:22.591 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2019-12-21T15:53:22.591 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:53:22.592 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:53:22.592 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:53:22.592 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.4.asok 2019-12-21T15:53:22.592 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:53:22.593 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.4.asok dump_scrub_reservations 2019-12-21T15:53:22.687 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=4 2019-12-21T15:53:22.687 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2019-12-21T15:53:22.688 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 14 + 4 2019-12-21T15:53:22.688 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=18 2019-12-21T15:53:22.689 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:53:22.690 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.5 2019-12-21T15:53:22.690 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2019-12-21T15:53:22.690 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2019-12-21T15:53:22.690 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:53:22.690 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:53:22.690 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:53:22.691 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.5.asok 2019-12-21T15:53:22.691 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:53:22.691 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.5.asok dump_scrub_reservations 2019-12-21T15:53:22.778 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:53:22.779 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 0, 2019-12-21T15:53:22.779 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 3, 2019-12-21T15:53:22.779 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:53:22.779 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:53:22.787 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:53:22.788 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.5 2019-12-21T15:53:22.788 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2019-12-21T15:53:22.788 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2019-12-21T15:53:22.788 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:53:22.788 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:53:22.789 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:53:22.789 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.5.asok 2019-12-21T15:53:22.789 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:53:22.789 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.5.asok dump_scrub_reservations 2019-12-21T15:53:22.883 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=3 2019-12-21T15:53:22.883 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2019-12-21T15:53:22.883 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 18 + 3 2019-12-21T15:53:22.884 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=21 2019-12-21T15:53:22.885 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:133: TEST_recover_unexpected: '[' 21 = 0 ']' 2019-12-21T15:53:22.885 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:136: TEST_recover_unexpected: '[' 0 -gt 3 ']' 2019-12-21T15:53:22.885 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: expr 175 + 21 2019-12-21T15:53:22.886 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: total=196 2019-12-21T15:53:22.887 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: expr 2 '*' 2 2019-12-21T15:53:22.888 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: sleep 4 2019-12-21T15:53:26.889 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:120: TEST_recover_unexpected: true 2019-12-21T15:53:26.889 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:122: TEST_recover_unexpected: pass=0 2019-12-21T15:53:26.891 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: expr 6 - 1 2019-12-21T15:53:26.891 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: seq 0 5 2019-12-21T15:53:26.892 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:53:26.894 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.0 2019-12-21T15:53:26.894 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2019-12-21T15:53:26.894 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2019-12-21T15:53:26.894 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:53:26.894 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:53:26.894 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:53:26.895 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.0.asok 2019-12-21T15:53:26.895 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:53:26.895 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.0.asok dump_scrub_reservations 2019-12-21T15:53:26.991 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:53:26.991 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 0, 2019-12-21T15:53:26.991 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 4, 2019-12-21T15:53:26.992 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:53:26.992 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:53:27.000 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:53:27.000 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.0 2019-12-21T15:53:27.001 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2019-12-21T15:53:27.001 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2019-12-21T15:53:27.001 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:53:27.001 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:53:27.001 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:53:27.001 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.0.asok 2019-12-21T15:53:27.002 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:53:27.002 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.0.asok dump_scrub_reservations 2019-12-21T15:53:27.097 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=4 2019-12-21T15:53:27.097 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2019-12-21T15:53:27.098 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 0 + 4 2019-12-21T15:53:27.098 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=4 2019-12-21T15:53:27.099 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:53:27.100 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.1 2019-12-21T15:53:27.100 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2019-12-21T15:53:27.100 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2019-12-21T15:53:27.100 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:53:27.100 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:53:27.101 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:53:27.101 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.1.asok 2019-12-21T15:53:27.101 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:53:27.101 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.1.asok dump_scrub_reservations 2019-12-21T15:53:27.189 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:53:27.189 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 1, 2019-12-21T15:53:27.189 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 1, 2019-12-21T15:53:27.189 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:53:27.189 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:53:27.197 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:53:27.197 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.1 2019-12-21T15:53:27.197 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2019-12-21T15:53:27.198 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2019-12-21T15:53:27.198 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:53:27.198 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:53:27.198 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:53:27.198 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.1.asok 2019-12-21T15:53:27.198 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:53:27.199 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.1.asok dump_scrub_reservations 2019-12-21T15:53:27.293 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=2 2019-12-21T15:53:27.293 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 2 -gt 4 ']' 2019-12-21T15:53:27.294 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 4 + 2 2019-12-21T15:53:27.295 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=6 2019-12-21T15:53:27.295 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:53:27.296 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.2 2019-12-21T15:53:27.296 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2019-12-21T15:53:27.297 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2019-12-21T15:53:27.297 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:53:27.297 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:53:27.297 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:53:27.297 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.2.asok 2019-12-21T15:53:27.297 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:53:27.297 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.2.asok dump_scrub_reservations 2019-12-21T15:53:27.384 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:53:27.385 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 1, 2019-12-21T15:53:27.385 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 1, 2019-12-21T15:53:27.385 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:53:27.385 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:53:27.393 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:53:27.393 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.2 2019-12-21T15:53:27.393 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2019-12-21T15:53:27.394 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2019-12-21T15:53:27.394 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:53:27.394 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:53:27.394 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:53:27.394 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.2.asok 2019-12-21T15:53:27.394 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:53:27.394 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.2.asok dump_scrub_reservations 2019-12-21T15:53:27.488 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=2 2019-12-21T15:53:27.488 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 2 -gt 4 ']' 2019-12-21T15:53:27.489 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 6 + 2 2019-12-21T15:53:27.490 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=8 2019-12-21T15:53:27.490 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:53:27.491 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.3 2019-12-21T15:53:27.491 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2019-12-21T15:53:27.491 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2019-12-21T15:53:27.491 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:53:27.491 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:53:27.492 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:53:27.492 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.3.asok 2019-12-21T15:53:27.492 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:53:27.492 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.3.asok dump_scrub_reservations 2019-12-21T15:53:27.580 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:53:27.580 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 3, 2019-12-21T15:53:27.580 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 1, 2019-12-21T15:53:27.580 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:53:27.581 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:53:27.592 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:53:27.592 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.3 2019-12-21T15:53:27.593 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2019-12-21T15:53:27.593 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2019-12-21T15:53:27.593 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:53:27.593 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:53:27.593 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:53:27.593 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.3.asok 2019-12-21T15:53:27.593 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:53:27.594 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.3.asok dump_scrub_reservations 2019-12-21T15:53:27.685 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=4 2019-12-21T15:53:27.685 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2019-12-21T15:53:27.686 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 8 + 4 2019-12-21T15:53:27.686 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=12 2019-12-21T15:53:27.687 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:53:27.688 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.4 2019-12-21T15:53:27.688 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2019-12-21T15:53:27.688 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2019-12-21T15:53:27.688 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:53:27.688 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:53:27.689 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:53:27.689 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.4.asok 2019-12-21T15:53:27.689 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:53:27.690 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.4.asok dump_scrub_reservations 2019-12-21T15:53:27.780 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:53:27.780 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 0, 2019-12-21T15:53:27.780 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 2, 2019-12-21T15:53:27.780 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:53:27.780 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:53:27.789 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:53:27.789 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.4 2019-12-21T15:53:27.789 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2019-12-21T15:53:27.789 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2019-12-21T15:53:27.789 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:53:27.790 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:53:27.790 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:53:27.790 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.4.asok 2019-12-21T15:53:27.790 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:53:27.790 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.4.asok dump_scrub_reservations 2019-12-21T15:53:27.884 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=2 2019-12-21T15:53:27.885 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 2 -gt 4 ']' 2019-12-21T15:53:27.885 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 12 + 2 2019-12-21T15:53:27.886 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=14 2019-12-21T15:53:27.886 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:53:27.887 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.5 2019-12-21T15:53:27.887 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2019-12-21T15:53:27.887 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2019-12-21T15:53:27.887 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:53:27.888 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:53:27.888 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:53:27.888 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.5.asok 2019-12-21T15:53:27.888 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:53:27.889 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.5.asok dump_scrub_reservations 2019-12-21T15:53:27.975 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:53:27.976 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 1, 2019-12-21T15:53:27.976 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 3, 2019-12-21T15:53:27.976 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:53:27.976 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:53:27.984 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:53:27.984 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.5 2019-12-21T15:53:27.985 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2019-12-21T15:53:27.985 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2019-12-21T15:53:27.985 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:53:27.985 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:53:27.985 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:53:27.985 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.5.asok 2019-12-21T15:53:27.986 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:53:27.986 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.5.asok dump_scrub_reservations 2019-12-21T15:53:28.080 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=4 2019-12-21T15:53:28.080 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2019-12-21T15:53:28.080 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 14 + 4 2019-12-21T15:53:28.081 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=18 2019-12-21T15:53:28.082 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:133: TEST_recover_unexpected: '[' 18 = 0 ']' 2019-12-21T15:53:28.082 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:136: TEST_recover_unexpected: '[' 0 -gt 3 ']' 2019-12-21T15:53:28.082 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: expr 196 + 18 2019-12-21T15:53:28.083 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: total=214 2019-12-21T15:53:28.083 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: expr 2 '*' 2 2019-12-21T15:53:28.085 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: sleep 4 2019-12-21T15:53:32.086 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:120: TEST_recover_unexpected: true 2019-12-21T15:53:32.086 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:122: TEST_recover_unexpected: pass=0 2019-12-21T15:53:32.087 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: expr 6 - 1 2019-12-21T15:53:32.089 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: seq 0 5 2019-12-21T15:53:32.089 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:53:32.091 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.0 2019-12-21T15:53:32.091 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2019-12-21T15:53:32.091 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2019-12-21T15:53:32.092 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:53:32.092 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:53:32.092 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:53:32.092 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.0.asok 2019-12-21T15:53:32.093 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:53:32.093 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.0.asok dump_scrub_reservations 2019-12-21T15:53:32.189 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:53:32.190 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 0, 2019-12-21T15:53:32.190 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 3, 2019-12-21T15:53:32.190 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:53:32.190 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:53:32.198 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:53:32.199 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.0 2019-12-21T15:53:32.199 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2019-12-21T15:53:32.199 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2019-12-21T15:53:32.199 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:53:32.199 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:53:32.199 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:53:32.199 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.0.asok 2019-12-21T15:53:32.200 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:53:32.200 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.0.asok dump_scrub_reservations 2019-12-21T15:53:32.294 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=3 2019-12-21T15:53:32.295 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2019-12-21T15:53:32.295 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 0 + 3 2019-12-21T15:53:32.295 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=3 2019-12-21T15:53:32.295 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:53:32.297 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.1 2019-12-21T15:53:32.297 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2019-12-21T15:53:32.297 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2019-12-21T15:53:32.297 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:53:32.297 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:53:32.298 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:53:32.298 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.1.asok 2019-12-21T15:53:32.298 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:53:32.298 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.1.asok dump_scrub_reservations 2019-12-21T15:53:32.390 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:53:32.391 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 1, 2019-12-21T15:53:32.391 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 3, 2019-12-21T15:53:32.391 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:53:32.391 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:53:32.395 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:53:32.395 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.1 2019-12-21T15:53:32.396 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2019-12-21T15:53:32.396 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2019-12-21T15:53:32.396 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:53:32.396 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:53:32.396 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:53:32.397 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.1.asok 2019-12-21T15:53:32.397 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:53:32.397 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.1.asok dump_scrub_reservations 2019-12-21T15:53:32.490 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=4 2019-12-21T15:53:32.491 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2019-12-21T15:53:32.491 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 3 + 4 2019-12-21T15:53:32.492 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=7 2019-12-21T15:53:32.492 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:53:32.495 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.2 2019-12-21T15:53:32.495 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2019-12-21T15:53:32.496 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2019-12-21T15:53:32.496 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:53:32.497 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:53:32.497 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:53:32.497 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.2.asok 2019-12-21T15:53:32.497 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:53:32.497 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.2.asok dump_scrub_reservations 2019-12-21T15:53:32.583 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:53:32.583 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 1, 2019-12-21T15:53:32.583 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 1, 2019-12-21T15:53:32.584 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:53:32.584 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:53:32.591 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:53:32.591 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.2 2019-12-21T15:53:32.592 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2019-12-21T15:53:32.593 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2019-12-21T15:53:32.594 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:53:32.594 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:53:32.594 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:53:32.594 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.2.asok 2019-12-21T15:53:32.594 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:53:32.594 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.2.asok dump_scrub_reservations 2019-12-21T15:53:32.688 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=2 2019-12-21T15:53:32.688 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 2 -gt 4 ']' 2019-12-21T15:53:32.689 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 7 + 2 2019-12-21T15:53:32.689 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=9 2019-12-21T15:53:32.690 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:53:32.691 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.3 2019-12-21T15:53:32.691 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2019-12-21T15:53:32.691 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2019-12-21T15:53:32.691 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:53:32.691 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:53:32.691 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:53:32.692 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.3.asok 2019-12-21T15:53:32.692 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:53:32.692 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.3.asok dump_scrub_reservations 2019-12-21T15:53:32.780 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:53:32.780 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 1, 2019-12-21T15:53:32.780 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 2, 2019-12-21T15:53:32.780 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:53:32.780 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:53:32.789 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:53:32.789 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.3 2019-12-21T15:53:32.789 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2019-12-21T15:53:32.789 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2019-12-21T15:53:32.790 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:53:32.790 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:53:32.790 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:53:32.790 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.3.asok 2019-12-21T15:53:32.790 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:53:32.790 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.3.asok dump_scrub_reservations 2019-12-21T15:53:32.885 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=3 2019-12-21T15:53:32.885 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2019-12-21T15:53:32.888 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 9 + 3 2019-12-21T15:53:32.888 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=12 2019-12-21T15:53:32.888 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:53:32.888 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.4 2019-12-21T15:53:32.889 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2019-12-21T15:53:32.889 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2019-12-21T15:53:32.889 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:53:32.889 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:53:32.889 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:53:32.889 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.4.asok 2019-12-21T15:53:32.889 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:53:32.890 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.4.asok dump_scrub_reservations 2019-12-21T15:53:32.982 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:53:32.983 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 1, 2019-12-21T15:53:32.983 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 1, 2019-12-21T15:53:32.983 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:53:32.983 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:53:32.985 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:53:32.985 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.4 2019-12-21T15:53:32.985 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2019-12-21T15:53:32.985 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2019-12-21T15:53:32.986 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:53:32.986 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:53:32.986 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:53:32.986 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.4.asok 2019-12-21T15:53:32.987 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:53:32.987 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.4.asok dump_scrub_reservations 2019-12-21T15:53:33.081 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=2 2019-12-21T15:53:33.082 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 2 -gt 4 ']' 2019-12-21T15:53:33.082 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 12 + 2 2019-12-21T15:53:33.083 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=14 2019-12-21T15:53:33.083 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:53:33.084 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.5 2019-12-21T15:53:33.084 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2019-12-21T15:53:33.085 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2019-12-21T15:53:33.085 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:53:33.085 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:53:33.085 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:53:33.085 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.5.asok 2019-12-21T15:53:33.085 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:53:33.086 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.5.asok dump_scrub_reservations 2019-12-21T15:53:33.172 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:53:33.173 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 2, 2019-12-21T15:53:33.173 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 2, 2019-12-21T15:53:33.173 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:53:33.173 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:53:33.181 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:53:33.182 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.5 2019-12-21T15:53:33.182 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2019-12-21T15:53:33.182 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2019-12-21T15:53:33.182 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:53:33.182 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:53:33.182 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:53:33.182 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.5.asok 2019-12-21T15:53:33.183 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:53:33.183 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.5.asok dump_scrub_reservations 2019-12-21T15:53:33.279 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=4 2019-12-21T15:53:33.279 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2019-12-21T15:53:33.279 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 14 + 4 2019-12-21T15:53:33.280 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=18 2019-12-21T15:53:33.280 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:133: TEST_recover_unexpected: '[' 18 = 0 ']' 2019-12-21T15:53:33.280 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:136: TEST_recover_unexpected: '[' 0 -gt 3 ']' 2019-12-21T15:53:33.281 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: expr 214 + 18 2019-12-21T15:53:33.281 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: total=232 2019-12-21T15:53:33.282 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: expr 2 '*' 2 2019-12-21T15:53:33.282 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: sleep 4 2019-12-21T15:53:37.284 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:120: TEST_recover_unexpected: true 2019-12-21T15:53:37.284 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:122: TEST_recover_unexpected: pass=0 2019-12-21T15:53:37.285 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: expr 6 - 1 2019-12-21T15:53:37.287 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: seq 0 5 2019-12-21T15:53:37.287 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:53:37.289 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.0 2019-12-21T15:53:37.289 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2019-12-21T15:53:37.289 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2019-12-21T15:53:37.289 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:53:37.290 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:53:37.290 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:53:37.290 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.0.asok 2019-12-21T15:53:37.291 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:53:37.291 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.0.asok dump_scrub_reservations 2019-12-21T15:53:37.386 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:53:37.387 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 0, 2019-12-21T15:53:37.387 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 3, 2019-12-21T15:53:37.387 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:53:37.387 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:53:37.396 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:53:37.396 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.0 2019-12-21T15:53:37.396 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2019-12-21T15:53:37.396 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2019-12-21T15:53:37.396 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:53:37.397 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:53:37.397 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:53:37.397 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.0.asok 2019-12-21T15:53:37.398 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:53:37.398 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.0.asok dump_scrub_reservations 2019-12-21T15:53:37.492 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=3 2019-12-21T15:53:37.493 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2019-12-21T15:53:37.493 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 0 + 3 2019-12-21T15:53:37.494 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=3 2019-12-21T15:53:37.494 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:53:37.495 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.1 2019-12-21T15:53:37.496 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2019-12-21T15:53:37.496 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2019-12-21T15:53:37.496 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:53:37.496 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:53:37.496 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:53:37.496 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.1.asok 2019-12-21T15:53:37.496 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:53:37.497 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.1.asok dump_scrub_reservations 2019-12-21T15:53:37.583 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:53:37.584 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 1, 2019-12-21T15:53:37.584 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 3, 2019-12-21T15:53:37.584 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:53:37.584 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:53:37.592 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:53:37.592 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.1 2019-12-21T15:53:37.593 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2019-12-21T15:53:37.593 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2019-12-21T15:53:37.593 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:53:37.593 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:53:37.593 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:53:37.593 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.1.asok 2019-12-21T15:53:37.593 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:53:37.594 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.1.asok dump_scrub_reservations 2019-12-21T15:53:37.687 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=4 2019-12-21T15:53:37.688 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2019-12-21T15:53:37.688 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 3 + 4 2019-12-21T15:53:37.689 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=7 2019-12-21T15:53:37.689 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:53:37.690 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.2 2019-12-21T15:53:37.690 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2019-12-21T15:53:37.690 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2019-12-21T15:53:37.691 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:53:37.691 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:53:37.691 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:53:37.691 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.2.asok 2019-12-21T15:53:37.692 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:53:37.692 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.2.asok dump_scrub_reservations 2019-12-21T15:53:37.778 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:53:37.779 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 1, 2019-12-21T15:53:37.779 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 1, 2019-12-21T15:53:37.779 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:53:37.779 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:53:37.788 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:53:37.788 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.2 2019-12-21T15:53:37.788 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2019-12-21T15:53:37.788 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2019-12-21T15:53:37.788 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:53:37.788 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:53:37.789 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:53:37.789 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.2.asok 2019-12-21T15:53:37.789 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:53:37.789 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.2.asok dump_scrub_reservations 2019-12-21T15:53:37.883 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=2 2019-12-21T15:53:37.883 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 2 -gt 4 ']' 2019-12-21T15:53:37.884 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 7 + 2 2019-12-21T15:53:37.884 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=9 2019-12-21T15:53:37.885 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:53:37.886 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.3 2019-12-21T15:53:37.886 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2019-12-21T15:53:37.886 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2019-12-21T15:53:37.886 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:53:37.886 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:53:37.887 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:53:37.887 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.3.asok 2019-12-21T15:53:37.887 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:53:37.887 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.3.asok dump_scrub_reservations 2019-12-21T15:53:37.975 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:53:37.975 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 1, 2019-12-21T15:53:37.975 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 2, 2019-12-21T15:53:37.975 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:53:37.975 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:53:37.984 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:53:37.984 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.3 2019-12-21T15:53:37.984 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2019-12-21T15:53:37.984 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2019-12-21T15:53:37.984 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:53:37.985 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:53:37.985 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:53:37.985 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.3.asok 2019-12-21T15:53:37.985 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:53:37.985 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.3.asok dump_scrub_reservations 2019-12-21T15:53:38.080 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=3 2019-12-21T15:53:38.080 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2019-12-21T15:53:38.080 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 9 + 3 2019-12-21T15:53:38.081 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=12 2019-12-21T15:53:38.081 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:53:38.083 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.4 2019-12-21T15:53:38.083 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2019-12-21T15:53:38.083 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2019-12-21T15:53:38.083 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:53:38.083 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:53:38.083 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:53:38.084 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.4.asok 2019-12-21T15:53:38.084 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:53:38.084 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.4.asok dump_scrub_reservations 2019-12-21T15:53:38.172 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:53:38.172 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 1, 2019-12-21T15:53:38.173 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 1, 2019-12-21T15:53:38.173 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:53:38.173 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:53:38.181 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:53:38.181 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.4 2019-12-21T15:53:38.181 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2019-12-21T15:53:38.181 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2019-12-21T15:53:38.182 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:53:38.182 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:53:38.182 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:53:38.182 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.4.asok 2019-12-21T15:53:38.182 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:53:38.182 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.4.asok dump_scrub_reservations 2019-12-21T15:53:38.277 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=2 2019-12-21T15:53:38.277 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 2 -gt 4 ']' 2019-12-21T15:53:38.278 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 12 + 2 2019-12-21T15:53:38.279 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=14 2019-12-21T15:53:38.279 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:53:38.280 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.5 2019-12-21T15:53:38.280 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2019-12-21T15:53:38.280 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2019-12-21T15:53:38.280 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:53:38.281 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:53:38.281 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:53:38.281 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.5.asok 2019-12-21T15:53:38.281 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:53:38.282 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.5.asok dump_scrub_reservations 2019-12-21T15:53:38.370 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:53:38.371 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 2, 2019-12-21T15:53:38.371 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 2, 2019-12-21T15:53:38.371 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:53:38.371 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:53:38.379 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:53:38.379 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.5 2019-12-21T15:53:38.380 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2019-12-21T15:53:38.380 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2019-12-21T15:53:38.380 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:53:38.380 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:53:38.380 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:53:38.380 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.5.asok 2019-12-21T15:53:38.381 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:53:38.381 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.5.asok dump_scrub_reservations 2019-12-21T15:53:38.476 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=4 2019-12-21T15:53:38.476 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2019-12-21T15:53:38.476 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 14 + 4 2019-12-21T15:53:38.477 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=18 2019-12-21T15:53:38.477 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:133: TEST_recover_unexpected: '[' 18 = 0 ']' 2019-12-21T15:53:38.477 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:136: TEST_recover_unexpected: '[' 0 -gt 3 ']' 2019-12-21T15:53:38.478 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: expr 232 + 18 2019-12-21T15:53:38.478 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: total=250 2019-12-21T15:53:38.479 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: expr 2 '*' 2 2019-12-21T15:53:38.480 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: sleep 4 2019-12-21T15:53:42.481 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:120: TEST_recover_unexpected: true 2019-12-21T15:53:42.482 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:122: TEST_recover_unexpected: pass=0 2019-12-21T15:53:42.483 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: expr 6 - 1 2019-12-21T15:53:42.484 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: seq 0 5 2019-12-21T15:53:42.485 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:53:42.486 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.0 2019-12-21T15:53:42.487 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2019-12-21T15:53:42.487 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2019-12-21T15:53:42.487 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:53:42.487 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:53:42.487 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:53:42.487 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.0.asok 2019-12-21T15:53:42.488 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:53:42.488 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.0.asok dump_scrub_reservations 2019-12-21T15:53:42.584 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:53:42.584 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 0, 2019-12-21T15:53:42.585 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 3, 2019-12-21T15:53:42.585 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:53:42.585 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:53:42.593 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:53:42.594 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.0 2019-12-21T15:53:42.594 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2019-12-21T15:53:42.594 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2019-12-21T15:53:42.594 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:53:42.594 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:53:42.594 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:53:42.595 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.0.asok 2019-12-21T15:53:42.595 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:53:42.595 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.0.asok dump_scrub_reservations 2019-12-21T15:53:42.692 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=3 2019-12-21T15:53:42.692 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2019-12-21T15:53:42.693 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 0 + 3 2019-12-21T15:53:42.693 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=3 2019-12-21T15:53:42.694 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:53:42.695 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.1 2019-12-21T15:53:42.695 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2019-12-21T15:53:42.695 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2019-12-21T15:53:42.695 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:53:42.695 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:53:42.695 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:53:42.696 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.1.asok 2019-12-21T15:53:42.696 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:53:42.696 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.1.asok dump_scrub_reservations 2019-12-21T15:53:42.782 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:53:42.783 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 1, 2019-12-21T15:53:42.783 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 3, 2019-12-21T15:53:42.783 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:53:42.783 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:53:42.791 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:53:42.791 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.1 2019-12-21T15:53:42.791 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2019-12-21T15:53:42.792 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2019-12-21T15:53:42.792 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:53:42.792 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:53:42.792 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:53:42.792 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.1.asok 2019-12-21T15:53:42.793 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:53:42.793 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.1.asok dump_scrub_reservations 2019-12-21T15:53:42.887 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=4 2019-12-21T15:53:42.887 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2019-12-21T15:53:42.887 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 3 + 4 2019-12-21T15:53:42.888 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=7 2019-12-21T15:53:42.889 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:53:42.890 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.2 2019-12-21T15:53:42.890 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2019-12-21T15:53:42.890 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2019-12-21T15:53:42.890 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:53:42.890 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:53:42.891 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:53:42.891 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.2.asok 2019-12-21T15:53:42.891 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:53:42.891 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.2.asok dump_scrub_reservations 2019-12-21T15:53:42.978 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:53:42.978 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 1, 2019-12-21T15:53:42.978 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 1, 2019-12-21T15:53:42.978 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:53:42.978 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:53:42.987 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:53:42.987 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.2 2019-12-21T15:53:42.987 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2019-12-21T15:53:42.987 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2019-12-21T15:53:42.987 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:53:42.987 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:53:42.988 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:53:42.988 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.2.asok 2019-12-21T15:53:42.988 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:53:42.988 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.2.asok dump_scrub_reservations 2019-12-21T15:53:43.082 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=2 2019-12-21T15:53:43.082 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 2 -gt 4 ']' 2019-12-21T15:53:43.083 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 7 + 2 2019-12-21T15:53:43.084 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=9 2019-12-21T15:53:43.084 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:53:43.085 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.3 2019-12-21T15:53:43.085 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2019-12-21T15:53:43.085 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2019-12-21T15:53:43.085 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:53:43.086 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:53:43.086 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:53:43.086 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.3.asok 2019-12-21T15:53:43.086 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:53:43.087 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.3.asok dump_scrub_reservations 2019-12-21T15:53:43.174 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:53:43.174 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 1, 2019-12-21T15:53:43.174 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 2, 2019-12-21T15:53:43.174 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:53:43.174 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:53:43.182 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:53:43.182 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.3 2019-12-21T15:53:43.183 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2019-12-21T15:53:43.183 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2019-12-21T15:53:43.183 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:53:43.183 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:53:43.183 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:53:43.183 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.3.asok 2019-12-21T15:53:43.184 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:53:43.184 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.3.asok dump_scrub_reservations 2019-12-21T15:53:43.279 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=3 2019-12-21T15:53:43.279 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2019-12-21T15:53:43.280 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 9 + 3 2019-12-21T15:53:43.281 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=12 2019-12-21T15:53:43.281 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:53:43.282 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.4 2019-12-21T15:53:43.282 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2019-12-21T15:53:43.282 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2019-12-21T15:53:43.283 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:53:43.283 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:53:43.283 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:53:43.283 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.4.asok 2019-12-21T15:53:43.283 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:53:43.283 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.4.asok dump_scrub_reservations 2019-12-21T15:53:43.371 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:53:43.371 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 1, 2019-12-21T15:53:43.371 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 1, 2019-12-21T15:53:43.371 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:53:43.371 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:53:43.379 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:53:43.380 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.4 2019-12-21T15:53:43.380 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2019-12-21T15:53:43.380 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2019-12-21T15:53:43.380 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:53:43.380 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:53:43.380 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:53:43.381 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.4.asok 2019-12-21T15:53:43.381 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:53:43.381 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.4.asok dump_scrub_reservations 2019-12-21T15:53:43.476 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=2 2019-12-21T15:53:43.476 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 2 -gt 4 ']' 2019-12-21T15:53:43.477 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 12 + 2 2019-12-21T15:53:43.477 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=14 2019-12-21T15:53:43.478 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:53:43.479 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.5 2019-12-21T15:53:43.479 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2019-12-21T15:53:43.479 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2019-12-21T15:53:43.479 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:53:43.480 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:53:43.480 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:53:43.480 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.5.asok 2019-12-21T15:53:43.480 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:53:43.480 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.5.asok dump_scrub_reservations 2019-12-21T15:53:43.568 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:53:43.569 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 2, 2019-12-21T15:53:43.569 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 2, 2019-12-21T15:53:43.569 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:53:43.569 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:53:43.577 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:53:43.577 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.5 2019-12-21T15:53:43.577 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2019-12-21T15:53:43.578 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2019-12-21T15:53:43.578 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:53:43.578 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:53:43.578 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:53:43.578 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.5.asok 2019-12-21T15:53:43.578 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:53:43.578 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.5.asok dump_scrub_reservations 2019-12-21T15:53:43.672 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=4 2019-12-21T15:53:43.673 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2019-12-21T15:53:43.673 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 14 + 4 2019-12-21T15:53:43.674 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=18 2019-12-21T15:53:43.674 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:133: TEST_recover_unexpected: '[' 18 = 0 ']' 2019-12-21T15:53:43.674 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:136: TEST_recover_unexpected: '[' 0 -gt 3 ']' 2019-12-21T15:53:43.674 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: expr 250 + 18 2019-12-21T15:53:43.675 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: total=268 2019-12-21T15:53:43.676 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: expr 2 '*' 2 2019-12-21T15:53:43.677 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: sleep 4 2019-12-21T15:53:47.678 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:120: TEST_recover_unexpected: true 2019-12-21T15:53:47.679 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:122: TEST_recover_unexpected: pass=0 2019-12-21T15:53:47.679 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: expr 6 - 1 2019-12-21T15:53:47.681 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: seq 0 5 2019-12-21T15:53:47.682 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:53:47.683 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.0 2019-12-21T15:53:47.683 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2019-12-21T15:53:47.684 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2019-12-21T15:53:47.684 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:53:47.684 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:53:47.684 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:53:47.685 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.0.asok 2019-12-21T15:53:47.685 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:53:47.685 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.0.asok dump_scrub_reservations 2019-12-21T15:53:47.784 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:53:47.784 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 0, 2019-12-21T15:53:47.784 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 3, 2019-12-21T15:53:47.785 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:53:47.785 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:53:47.794 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:53:47.794 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.0 2019-12-21T15:53:47.794 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2019-12-21T15:53:47.795 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2019-12-21T15:53:47.795 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:53:47.795 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:53:47.795 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:53:47.795 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.0.asok 2019-12-21T15:53:47.795 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:53:47.795 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.0.asok dump_scrub_reservations 2019-12-21T15:53:47.890 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=2 2019-12-21T15:53:47.891 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 2 -gt 4 ']' 2019-12-21T15:53:47.891 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 0 + 2 2019-12-21T15:53:47.892 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=2 2019-12-21T15:53:47.892 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:53:47.893 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.1 2019-12-21T15:53:47.893 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2019-12-21T15:53:47.893 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2019-12-21T15:53:47.894 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:53:47.894 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:53:47.894 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:53:47.894 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.1.asok 2019-12-21T15:53:47.894 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:53:47.895 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.1.asok dump_scrub_reservations 2019-12-21T15:53:47.981 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:53:47.982 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 1, 2019-12-21T15:53:47.982 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 3, 2019-12-21T15:53:47.982 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:53:47.982 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:53:47.990 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:53:47.990 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.1 2019-12-21T15:53:47.991 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2019-12-21T15:53:47.991 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2019-12-21T15:53:47.991 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:53:47.991 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:53:47.991 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:53:47.991 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.1.asok 2019-12-21T15:53:47.992 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:53:47.992 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.1.asok dump_scrub_reservations 2019-12-21T15:53:48.085 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=4 2019-12-21T15:53:48.085 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2019-12-21T15:53:48.085 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 2 + 4 2019-12-21T15:53:48.086 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=6 2019-12-21T15:53:48.086 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:53:48.088 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.2 2019-12-21T15:53:48.088 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2019-12-21T15:53:48.088 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2019-12-21T15:53:48.088 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:53:48.088 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:53:48.089 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:53:48.089 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.2.asok 2019-12-21T15:53:48.089 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:53:48.089 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.2.asok dump_scrub_reservations 2019-12-21T15:53:48.177 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:53:48.177 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 0, 2019-12-21T15:53:48.177 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 1, 2019-12-21T15:53:48.177 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:53:48.177 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:53:48.185 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:53:48.185 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.2 2019-12-21T15:53:48.185 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2019-12-21T15:53:48.186 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2019-12-21T15:53:48.186 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:53:48.186 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:53:48.186 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:53:48.186 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.2.asok 2019-12-21T15:53:48.187 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:53:48.187 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.2.asok dump_scrub_reservations 2019-12-21T15:53:48.280 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=1 2019-12-21T15:53:48.280 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 1 -gt 4 ']' 2019-12-21T15:53:48.281 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 6 + 1 2019-12-21T15:53:48.282 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=7 2019-12-21T15:53:48.282 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:53:48.283 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.3 2019-12-21T15:53:48.283 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2019-12-21T15:53:48.283 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2019-12-21T15:53:48.284 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:53:48.284 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:53:48.284 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:53:48.284 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.3.asok 2019-12-21T15:53:48.284 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:53:48.284 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.3.asok dump_scrub_reservations 2019-12-21T15:53:48.371 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:53:48.372 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 0, 2019-12-21T15:53:48.372 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 2, 2019-12-21T15:53:48.372 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:53:48.372 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:53:48.380 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:53:48.380 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.3 2019-12-21T15:53:48.380 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2019-12-21T15:53:48.381 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2019-12-21T15:53:48.381 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:53:48.381 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:53:48.381 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:53:48.381 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.3.asok 2019-12-21T15:53:48.381 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:53:48.381 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.3.asok dump_scrub_reservations 2019-12-21T15:53:48.476 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=2 2019-12-21T15:53:48.476 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 2 -gt 4 ']' 2019-12-21T15:53:48.477 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 7 + 2 2019-12-21T15:53:48.477 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=9 2019-12-21T15:53:48.477 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:53:48.479 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.4 2019-12-21T15:53:48.479 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2019-12-21T15:53:48.479 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2019-12-21T15:53:48.479 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:53:48.479 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:53:48.479 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:53:48.480 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.4.asok 2019-12-21T15:53:48.480 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:53:48.480 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.4.asok dump_scrub_reservations 2019-12-21T15:53:48.567 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:53:48.567 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 1, 2019-12-21T15:53:48.568 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 2, 2019-12-21T15:53:48.568 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:53:48.568 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:53:48.576 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:53:48.576 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.4 2019-12-21T15:53:48.577 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2019-12-21T15:53:48.577 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2019-12-21T15:53:48.577 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:53:48.577 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:53:48.577 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:53:48.577 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.4.asok 2019-12-21T15:53:48.577 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:53:48.578 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.4.asok dump_scrub_reservations 2019-12-21T15:53:48.673 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=4 2019-12-21T15:53:48.674 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2019-12-21T15:53:48.674 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 9 + 4 2019-12-21T15:53:48.675 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=13 2019-12-21T15:53:48.675 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:53:48.676 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.5 2019-12-21T15:53:48.676 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2019-12-21T15:53:48.676 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2019-12-21T15:53:48.677 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:53:48.677 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:53:48.677 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:53:48.677 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.5.asok 2019-12-21T15:53:48.677 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:53:48.678 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.5.asok dump_scrub_reservations 2019-12-21T15:53:48.765 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:53:48.765 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 4, 2019-12-21T15:53:48.766 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 0, 2019-12-21T15:53:48.766 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:53:48.766 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:53:48.774 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:53:48.774 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.5 2019-12-21T15:53:48.774 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2019-12-21T15:53:48.774 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2019-12-21T15:53:48.775 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:53:48.775 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:53:48.775 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:53:48.775 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.5.asok 2019-12-21T15:53:48.775 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:53:48.775 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.5.asok dump_scrub_reservations 2019-12-21T15:53:48.870 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=4 2019-12-21T15:53:48.870 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2019-12-21T15:53:48.871 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 13 + 4 2019-12-21T15:53:48.871 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=17 2019-12-21T15:53:48.872 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:133: TEST_recover_unexpected: '[' 17 = 0 ']' 2019-12-21T15:53:48.872 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:136: TEST_recover_unexpected: '[' 0 -gt 3 ']' 2019-12-21T15:53:48.872 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: expr 268 + 17 2019-12-21T15:53:48.873 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: total=285 2019-12-21T15:53:48.874 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: expr 2 '*' 2 2019-12-21T15:53:48.874 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: sleep 4 2019-12-21T15:53:52.876 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:120: TEST_recover_unexpected: true 2019-12-21T15:53:52.876 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:122: TEST_recover_unexpected: pass=0 2019-12-21T15:53:52.885 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: expr 6 - 1 2019-12-21T15:53:52.885 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: seq 0 5 2019-12-21T15:53:52.885 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:53:52.886 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.0 2019-12-21T15:53:52.886 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2019-12-21T15:53:52.886 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2019-12-21T15:53:52.886 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:53:52.886 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:53:52.886 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:53:52.887 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.0.asok 2019-12-21T15:53:52.887 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:53:52.887 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.0.asok dump_scrub_reservations 2019-12-21T15:53:52.979 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:53:52.979 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 1, 2019-12-21T15:53:52.980 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 2, 2019-12-21T15:53:52.980 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:53:52.980 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:53:52.989 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:53:52.989 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.0 2019-12-21T15:53:52.989 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2019-12-21T15:53:52.989 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2019-12-21T15:53:52.989 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:53:52.989 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:53:52.990 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:53:52.990 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.0.asok 2019-12-21T15:53:52.990 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:53:52.990 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.0.asok dump_scrub_reservations 2019-12-21T15:53:53.085 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=3 2019-12-21T15:53:53.085 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2019-12-21T15:53:53.086 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 0 + 3 2019-12-21T15:53:53.087 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=3 2019-12-21T15:53:53.087 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:53:53.088 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.1 2019-12-21T15:53:53.088 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2019-12-21T15:53:53.089 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2019-12-21T15:53:53.089 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:53:53.089 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:53:53.089 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:53:53.089 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.1.asok 2019-12-21T15:53:53.089 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:53:53.089 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.1.asok dump_scrub_reservations 2019-12-21T15:53:53.177 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:53:53.177 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 1, 2019-12-21T15:53:53.177 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 2, 2019-12-21T15:53:53.178 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:53:53.178 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:53:53.186 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:53:53.186 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.1 2019-12-21T15:53:53.186 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2019-12-21T15:53:53.186 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2019-12-21T15:53:53.187 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:53:53.187 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:53:53.187 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:53:53.187 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.1.asok 2019-12-21T15:53:53.187 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:53:53.187 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.1.asok dump_scrub_reservations 2019-12-21T15:53:53.282 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=3 2019-12-21T15:53:53.282 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2019-12-21T15:53:53.282 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 3 + 3 2019-12-21T15:53:53.283 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=6 2019-12-21T15:53:53.283 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:53:53.284 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.2 2019-12-21T15:53:53.285 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2019-12-21T15:53:53.285 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2019-12-21T15:53:53.285 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:53:53.285 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:53:53.285 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:53:53.285 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.2.asok 2019-12-21T15:53:53.286 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:53:53.286 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.2.asok dump_scrub_reservations 2019-12-21T15:53:53.374 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:53:53.374 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 0, 2019-12-21T15:53:53.374 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 1, 2019-12-21T15:53:53.374 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:53:53.374 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:53:53.382 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:53:53.382 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.2 2019-12-21T15:53:53.382 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2019-12-21T15:53:53.383 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2019-12-21T15:53:53.383 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:53:53.383 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:53:53.383 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:53:53.383 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.2.asok 2019-12-21T15:53:53.384 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:53:53.384 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.2.asok dump_scrub_reservations 2019-12-21T15:53:53.478 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=1 2019-12-21T15:53:53.478 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 1 -gt 4 ']' 2019-12-21T15:53:53.479 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 6 + 1 2019-12-21T15:53:53.480 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=7 2019-12-21T15:53:53.480 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:53:53.481 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.3 2019-12-21T15:53:53.481 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2019-12-21T15:53:53.481 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2019-12-21T15:53:53.482 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:53:53.482 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:53:53.482 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:53:53.482 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.3.asok 2019-12-21T15:53:53.482 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:53:53.482 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.3.asok dump_scrub_reservations 2019-12-21T15:53:53.573 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:53:53.574 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 1, 2019-12-21T15:53:53.574 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 2, 2019-12-21T15:53:53.574 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:53:53.574 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:53:53.579 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:53:53.579 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.3 2019-12-21T15:53:53.580 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2019-12-21T15:53:53.580 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2019-12-21T15:53:53.580 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:53:53.580 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:53:53.580 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:53:53.580 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.3.asok 2019-12-21T15:53:53.581 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:53:53.581 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.3.asok dump_scrub_reservations 2019-12-21T15:53:53.676 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=3 2019-12-21T15:53:53.676 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2019-12-21T15:53:53.676 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 7 + 3 2019-12-21T15:53:53.677 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=10 2019-12-21T15:53:53.677 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:53:53.678 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.4 2019-12-21T15:53:53.679 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2019-12-21T15:53:53.679 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2019-12-21T15:53:53.679 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:53:53.679 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:53:53.679 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:53:53.679 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.4.asok 2019-12-21T15:53:53.680 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:53:53.680 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.4.asok dump_scrub_reservations 2019-12-21T15:53:53.767 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:53:53.768 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 0, 2019-12-21T15:53:53.768 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 3, 2019-12-21T15:53:53.768 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:53:53.768 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:53:53.776 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:53:53.777 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.4 2019-12-21T15:53:53.777 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2019-12-21T15:53:53.777 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2019-12-21T15:53:53.777 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:53:53.777 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:53:53.777 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:53:53.778 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.4.asok 2019-12-21T15:53:53.778 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:53:53.778 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.4.asok dump_scrub_reservations 2019-12-21T15:53:53.871 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=3 2019-12-21T15:53:53.872 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2019-12-21T15:53:53.872 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 10 + 3 2019-12-21T15:53:53.873 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=13 2019-12-21T15:53:53.873 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:53:53.874 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.5 2019-12-21T15:53:53.874 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2019-12-21T15:53:53.875 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2019-12-21T15:53:53.875 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:53:53.875 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:53:53.875 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:53:53.875 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.5.asok 2019-12-21T15:53:53.875 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:53:53.876 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.5.asok dump_scrub_reservations 2019-12-21T15:53:53.963 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:53:53.963 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 3, 2019-12-21T15:53:53.963 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 1, 2019-12-21T15:53:53.963 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:53:53.964 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:53:53.972 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:53:53.972 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.5 2019-12-21T15:53:53.972 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2019-12-21T15:53:53.972 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2019-12-21T15:53:53.973 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:53:53.973 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:53:53.973 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:53:53.973 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.5.asok 2019-12-21T15:53:53.973 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:53:53.973 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.5.asok dump_scrub_reservations 2019-12-21T15:53:54.068 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=4 2019-12-21T15:53:54.069 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2019-12-21T15:53:54.069 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 13 + 4 2019-12-21T15:53:54.070 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=17 2019-12-21T15:53:54.071 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:133: TEST_recover_unexpected: '[' 17 = 0 ']' 2019-12-21T15:53:54.071 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:136: TEST_recover_unexpected: '[' 0 -gt 3 ']' 2019-12-21T15:53:54.071 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: expr 285 + 17 2019-12-21T15:53:54.072 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: total=302 2019-12-21T15:53:54.075 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: expr 2 '*' 2 2019-12-21T15:53:54.075 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: sleep 4 2019-12-21T15:53:58.074 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:120: TEST_recover_unexpected: true 2019-12-21T15:53:58.075 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:122: TEST_recover_unexpected: pass=0 2019-12-21T15:53:58.076 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: expr 6 - 1 2019-12-21T15:53:58.077 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: seq 0 5 2019-12-21T15:53:58.078 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:53:58.079 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.0 2019-12-21T15:53:58.080 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2019-12-21T15:53:58.080 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2019-12-21T15:53:58.080 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:53:58.080 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:53:58.080 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:53:58.081 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.0.asok 2019-12-21T15:53:58.081 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:53:58.083 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.0.asok dump_scrub_reservations 2019-12-21T15:53:58.178 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:53:58.178 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 1, 2019-12-21T15:53:58.178 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 3, 2019-12-21T15:53:58.178 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:53:58.178 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:53:58.187 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:53:58.187 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.0 2019-12-21T15:53:58.188 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2019-12-21T15:53:58.188 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2019-12-21T15:53:58.188 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:53:58.188 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:53:58.188 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:53:58.188 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.0.asok 2019-12-21T15:53:58.189 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:53:58.189 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.0.asok dump_scrub_reservations 2019-12-21T15:53:58.284 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=4 2019-12-21T15:53:58.284 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2019-12-21T15:53:58.284 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 0 + 4 2019-12-21T15:53:58.285 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=4 2019-12-21T15:53:58.285 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:53:58.287 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.1 2019-12-21T15:53:58.287 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2019-12-21T15:53:58.287 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2019-12-21T15:53:58.287 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:53:58.287 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:53:58.287 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:53:58.288 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.1.asok 2019-12-21T15:53:58.288 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:53:58.288 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.1.asok dump_scrub_reservations 2019-12-21T15:53:58.374 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:53:58.374 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 2, 2019-12-21T15:53:58.375 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 1, 2019-12-21T15:53:58.375 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:53:58.375 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:53:58.383 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:53:58.383 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.1 2019-12-21T15:53:58.383 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2019-12-21T15:53:58.384 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2019-12-21T15:53:58.384 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:53:58.384 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:53:58.384 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:53:58.384 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.1.asok 2019-12-21T15:53:58.384 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:53:58.385 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.1.asok dump_scrub_reservations 2019-12-21T15:53:58.478 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=3 2019-12-21T15:53:58.479 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2019-12-21T15:53:58.479 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 4 + 3 2019-12-21T15:53:58.480 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=7 2019-12-21T15:53:58.480 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:53:58.482 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.2 2019-12-21T15:53:58.483 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2019-12-21T15:53:58.483 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2019-12-21T15:53:58.483 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:53:58.483 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:53:58.483 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:53:58.484 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.2.asok 2019-12-21T15:53:58.484 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:53:58.485 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.2.asok dump_scrub_reservations 2019-12-21T15:53:58.570 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:53:58.570 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 0, 2019-12-21T15:53:58.570 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 1, 2019-12-21T15:53:58.570 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:53:58.570 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:53:58.579 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:53:58.580 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.2 2019-12-21T15:53:58.580 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2019-12-21T15:53:58.580 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2019-12-21T15:53:58.580 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:53:58.580 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:53:58.581 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:53:58.581 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.2.asok 2019-12-21T15:53:58.581 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:53:58.581 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.2.asok dump_scrub_reservations 2019-12-21T15:53:58.676 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=1 2019-12-21T15:53:58.677 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 1 -gt 4 ']' 2019-12-21T15:53:58.678 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 7 + 1 2019-12-21T15:53:58.678 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=8 2019-12-21T15:53:58.679 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:53:58.680 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.3 2019-12-21T15:53:58.680 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2019-12-21T15:53:58.680 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2019-12-21T15:53:58.680 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:53:58.681 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:53:58.681 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:53:58.681 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.3.asok 2019-12-21T15:53:58.681 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:53:58.681 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.3.asok dump_scrub_reservations 2019-12-21T15:53:58.768 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:53:58.768 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 1, 2019-12-21T15:53:58.768 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 1, 2019-12-21T15:53:58.768 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:53:58.769 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:53:58.777 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:53:58.778 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.3 2019-12-21T15:53:58.778 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2019-12-21T15:53:58.778 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2019-12-21T15:53:58.778 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:53:58.779 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:53:58.779 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:53:58.779 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.3.asok 2019-12-21T15:53:58.779 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:53:58.779 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.3.asok dump_scrub_reservations 2019-12-21T15:53:58.873 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=2 2019-12-21T15:53:58.873 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 2 -gt 4 ']' 2019-12-21T15:53:58.874 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 8 + 2 2019-12-21T15:53:58.874 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=10 2019-12-21T15:53:58.874 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:53:58.876 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.4 2019-12-21T15:53:58.876 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2019-12-21T15:53:58.876 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2019-12-21T15:53:58.876 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:53:58.877 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:53:58.877 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:53:58.877 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.4.asok 2019-12-21T15:53:58.877 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:53:58.877 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.4.asok dump_scrub_reservations 2019-12-21T15:53:58.964 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:53:58.964 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 0, 2019-12-21T15:53:58.964 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 4, 2019-12-21T15:53:58.964 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:53:58.964 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:53:58.972 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:53:58.973 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.4 2019-12-21T15:53:58.973 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2019-12-21T15:53:58.973 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2019-12-21T15:53:58.973 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:53:58.973 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:53:58.973 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:53:58.974 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.4.asok 2019-12-21T15:53:58.974 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:53:58.974 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.4.asok dump_scrub_reservations 2019-12-21T15:53:59.068 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=4 2019-12-21T15:53:59.068 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2019-12-21T15:53:59.068 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 10 + 4 2019-12-21T15:53:59.069 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=14 2019-12-21T15:53:59.069 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:53:59.070 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.5 2019-12-21T15:53:59.070 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2019-12-21T15:53:59.071 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2019-12-21T15:53:59.071 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:53:59.071 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:53:59.071 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:53:59.071 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.5.asok 2019-12-21T15:53:59.072 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:53:59.072 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.5.asok dump_scrub_reservations 2019-12-21T15:53:59.159 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:53:59.159 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 2, 2019-12-21T15:53:59.159 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 2, 2019-12-21T15:53:59.159 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:53:59.159 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:53:59.167 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:53:59.167 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.5 2019-12-21T15:53:59.168 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2019-12-21T15:53:59.168 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2019-12-21T15:53:59.168 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:53:59.168 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:53:59.168 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:53:59.168 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.5.asok 2019-12-21T15:53:59.169 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:53:59.169 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.5.asok dump_scrub_reservations 2019-12-21T15:53:59.264 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=4 2019-12-21T15:53:59.264 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2019-12-21T15:53:59.264 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 14 + 4 2019-12-21T15:53:59.265 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=18 2019-12-21T15:53:59.265 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:133: TEST_recover_unexpected: '[' 18 = 0 ']' 2019-12-21T15:53:59.265 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:136: TEST_recover_unexpected: '[' 0 -gt 3 ']' 2019-12-21T15:53:59.265 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: expr 302 + 18 2019-12-21T15:53:59.269 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: total=320 2019-12-21T15:53:59.269 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: expr 2 '*' 2 2019-12-21T15:53:59.269 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: sleep 4 2019-12-21T15:54:03.269 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:120: TEST_recover_unexpected: true 2019-12-21T15:54:03.270 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:122: TEST_recover_unexpected: pass=0 2019-12-21T15:54:03.270 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: expr 6 - 1 2019-12-21T15:54:03.271 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: seq 0 5 2019-12-21T15:54:03.276 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:54:03.276 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.0 2019-12-21T15:54:03.276 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2019-12-21T15:54:03.276 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2019-12-21T15:54:03.276 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:54:03.276 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:54:03.277 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:54:03.277 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.0.asok 2019-12-21T15:54:03.277 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:54:03.277 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.0.asok dump_scrub_reservations 2019-12-21T15:54:03.371 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:54:03.371 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 1, 2019-12-21T15:54:03.371 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 3, 2019-12-21T15:54:03.371 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:54:03.372 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:54:03.381 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:54:03.381 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.0 2019-12-21T15:54:03.381 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2019-12-21T15:54:03.381 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2019-12-21T15:54:03.381 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:54:03.381 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:54:03.382 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:54:03.382 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.0.asok 2019-12-21T15:54:03.382 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:54:03.382 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.0.asok dump_scrub_reservations 2019-12-21T15:54:03.477 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=4 2019-12-21T15:54:03.477 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2019-12-21T15:54:03.477 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 0 + 4 2019-12-21T15:54:03.478 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=4 2019-12-21T15:54:03.478 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:54:03.480 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.1 2019-12-21T15:54:03.480 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2019-12-21T15:54:03.480 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2019-12-21T15:54:03.480 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:54:03.480 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:54:03.481 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:54:03.481 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.1.asok 2019-12-21T15:54:03.481 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:54:03.481 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.1.asok dump_scrub_reservations 2019-12-21T15:54:03.568 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:54:03.569 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 2, 2019-12-21T15:54:03.569 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 1, 2019-12-21T15:54:03.569 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:54:03.569 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:54:03.579 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:54:03.579 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.1 2019-12-21T15:54:03.579 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2019-12-21T15:54:03.579 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2019-12-21T15:54:03.580 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:54:03.580 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:54:03.580 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:54:03.580 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.1.asok 2019-12-21T15:54:03.580 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:54:03.580 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.1.asok dump_scrub_reservations 2019-12-21T15:54:03.673 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=3 2019-12-21T15:54:03.673 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2019-12-21T15:54:03.674 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 4 + 3 2019-12-21T15:54:03.674 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=7 2019-12-21T15:54:03.674 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:54:03.676 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.2 2019-12-21T15:54:03.676 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2019-12-21T15:54:03.676 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2019-12-21T15:54:03.676 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:54:03.676 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:54:03.677 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:54:03.677 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.2.asok 2019-12-21T15:54:03.677 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:54:03.677 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.2.asok dump_scrub_reservations 2019-12-21T15:54:03.765 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:54:03.765 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 0, 2019-12-21T15:54:03.765 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 1, 2019-12-21T15:54:03.766 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:54:03.766 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:54:03.774 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:54:03.775 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.2 2019-12-21T15:54:03.775 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2019-12-21T15:54:03.775 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2019-12-21T15:54:03.775 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:54:03.775 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:54:03.775 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:54:03.775 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.2.asok 2019-12-21T15:54:03.776 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:54:03.776 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.2.asok dump_scrub_reservations 2019-12-21T15:54:03.869 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=1 2019-12-21T15:54:03.870 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 1 -gt 4 ']' 2019-12-21T15:54:03.870 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 7 + 1 2019-12-21T15:54:03.871 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=8 2019-12-21T15:54:03.871 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:54:03.872 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.3 2019-12-21T15:54:03.873 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2019-12-21T15:54:03.873 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2019-12-21T15:54:03.873 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:54:03.873 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:54:03.874 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:54:03.874 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.3.asok 2019-12-21T15:54:03.875 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:54:03.875 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.3.asok dump_scrub_reservations 2019-12-21T15:54:03.961 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:54:03.963 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 1, 2019-12-21T15:54:03.964 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 1, 2019-12-21T15:54:03.964 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:54:03.964 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:54:03.970 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:54:03.970 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.3 2019-12-21T15:54:03.970 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2019-12-21T15:54:03.970 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2019-12-21T15:54:03.970 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:54:03.970 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:54:03.970 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:54:03.971 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.3.asok 2019-12-21T15:54:03.971 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:54:03.971 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.3.asok dump_scrub_reservations 2019-12-21T15:54:04.065 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=2 2019-12-21T15:54:04.065 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 2 -gt 4 ']' 2019-12-21T15:54:04.065 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 8 + 2 2019-12-21T15:54:04.066 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=10 2019-12-21T15:54:04.067 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:54:04.068 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.4 2019-12-21T15:54:04.068 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2019-12-21T15:54:04.068 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2019-12-21T15:54:04.068 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:54:04.069 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:54:04.069 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:54:04.069 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.4.asok 2019-12-21T15:54:04.069 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:54:04.069 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.4.asok dump_scrub_reservations 2019-12-21T15:54:04.156 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:54:04.157 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 0, 2019-12-21T15:54:04.157 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 4, 2019-12-21T15:54:04.157 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:54:04.157 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:54:04.166 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:54:04.166 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.4 2019-12-21T15:54:04.166 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2019-12-21T15:54:04.166 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2019-12-21T15:54:04.166 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:54:04.167 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:54:04.167 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:54:04.167 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.4.asok 2019-12-21T15:54:04.167 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:54:04.167 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.4.asok dump_scrub_reservations 2019-12-21T15:54:04.262 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=4 2019-12-21T15:54:04.262 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2019-12-21T15:54:04.262 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 10 + 4 2019-12-21T15:54:04.263 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=14 2019-12-21T15:54:04.264 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:54:04.266 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.5 2019-12-21T15:54:04.266 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2019-12-21T15:54:04.266 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2019-12-21T15:54:04.266 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:54:04.266 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:54:04.267 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:54:04.267 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.5.asok 2019-12-21T15:54:04.267 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:54:04.267 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.5.asok dump_scrub_reservations 2019-12-21T15:54:04.354 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:54:04.354 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 2, 2019-12-21T15:54:04.354 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 2, 2019-12-21T15:54:04.354 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:54:04.354 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:54:04.362 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:54:04.362 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.5 2019-12-21T15:54:04.363 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2019-12-21T15:54:04.363 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2019-12-21T15:54:04.363 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:54:04.363 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:54:04.363 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:54:04.363 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.5.asok 2019-12-21T15:54:04.364 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:54:04.364 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.5.asok dump_scrub_reservations 2019-12-21T15:54:04.458 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=4 2019-12-21T15:54:04.458 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2019-12-21T15:54:04.459 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 14 + 4 2019-12-21T15:54:04.460 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=18 2019-12-21T15:54:04.460 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:133: TEST_recover_unexpected: '[' 18 = 0 ']' 2019-12-21T15:54:04.460 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:136: TEST_recover_unexpected: '[' 0 -gt 3 ']' 2019-12-21T15:54:04.461 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: expr 320 + 18 2019-12-21T15:54:04.461 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: total=338 2019-12-21T15:54:04.462 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: expr 2 '*' 2 2019-12-21T15:54:04.462 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: sleep 4 2019-12-21T15:54:08.464 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:120: TEST_recover_unexpected: true 2019-12-21T15:54:08.464 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:122: TEST_recover_unexpected: pass=0 2019-12-21T15:54:08.465 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: expr 6 - 1 2019-12-21T15:54:08.467 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: seq 0 5 2019-12-21T15:54:08.468 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:54:08.469 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.0 2019-12-21T15:54:08.469 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2019-12-21T15:54:08.469 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2019-12-21T15:54:08.470 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:54:08.470 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:54:08.470 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:54:08.470 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.0.asok 2019-12-21T15:54:08.470 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:54:08.471 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.0.asok dump_scrub_reservations 2019-12-21T15:54:08.568 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:54:08.569 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 1, 2019-12-21T15:54:08.569 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 3, 2019-12-21T15:54:08.569 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:54:08.570 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:54:08.578 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:54:08.578 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.0 2019-12-21T15:54:08.579 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2019-12-21T15:54:08.579 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2019-12-21T15:54:08.579 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:54:08.579 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:54:08.579 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:54:08.579 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.0.asok 2019-12-21T15:54:08.580 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:54:08.580 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.0.asok dump_scrub_reservations 2019-12-21T15:54:08.676 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=4 2019-12-21T15:54:08.676 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2019-12-21T15:54:08.677 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 0 + 4 2019-12-21T15:54:08.678 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=4 2019-12-21T15:54:08.678 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:54:08.680 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.1 2019-12-21T15:54:08.680 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2019-12-21T15:54:08.680 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2019-12-21T15:54:08.680 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:54:08.680 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:54:08.680 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:54:08.681 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.1.asok 2019-12-21T15:54:08.681 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:54:08.681 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.1.asok dump_scrub_reservations 2019-12-21T15:54:08.768 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:54:08.768 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 2, 2019-12-21T15:54:08.768 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 1, 2019-12-21T15:54:08.768 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:54:08.769 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:54:08.777 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:54:08.777 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.1 2019-12-21T15:54:08.777 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2019-12-21T15:54:08.777 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2019-12-21T15:54:08.777 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:54:08.778 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:54:08.778 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:54:08.778 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.1.asok 2019-12-21T15:54:08.778 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:54:08.778 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.1.asok dump_scrub_reservations 2019-12-21T15:54:08.873 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=3 2019-12-21T15:54:08.873 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2019-12-21T15:54:08.874 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 4 + 3 2019-12-21T15:54:08.874 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=7 2019-12-21T15:54:08.875 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:54:08.884 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.2 2019-12-21T15:54:08.884 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2019-12-21T15:54:08.884 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2019-12-21T15:54:08.884 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:54:08.884 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:54:08.884 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:54:08.885 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.2.asok 2019-12-21T15:54:08.885 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:54:08.885 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.2.asok dump_scrub_reservations 2019-12-21T15:54:08.965 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:54:08.965 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 0, 2019-12-21T15:54:08.965 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 1, 2019-12-21T15:54:08.965 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:54:08.965 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:54:08.973 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:54:08.973 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.2 2019-12-21T15:54:08.974 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2019-12-21T15:54:08.974 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2019-12-21T15:54:08.974 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:54:08.974 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:54:08.974 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:54:08.974 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.2.asok 2019-12-21T15:54:08.975 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:54:08.975 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.2.asok dump_scrub_reservations 2019-12-21T15:54:09.069 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=1 2019-12-21T15:54:09.069 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 1 -gt 4 ']' 2019-12-21T15:54:09.069 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 7 + 1 2019-12-21T15:54:09.071 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=8 2019-12-21T15:54:09.071 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:54:09.073 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.3 2019-12-21T15:54:09.073 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2019-12-21T15:54:09.073 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2019-12-21T15:54:09.073 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:54:09.074 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:54:09.074 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:54:09.074 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.3.asok 2019-12-21T15:54:09.074 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:54:09.074 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.3.asok dump_scrub_reservations 2019-12-21T15:54:09.160 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:54:09.161 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 1, 2019-12-21T15:54:09.161 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 1, 2019-12-21T15:54:09.161 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:54:09.161 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:54:09.175 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:54:09.175 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.3 2019-12-21T15:54:09.177 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2019-12-21T15:54:09.177 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2019-12-21T15:54:09.178 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:54:09.178 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:54:09.178 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:54:09.178 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.3.asok 2019-12-21T15:54:09.179 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:54:09.179 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.3.asok dump_scrub_reservations 2019-12-21T15:54:09.266 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=2 2019-12-21T15:54:09.266 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 2 -gt 4 ']' 2019-12-21T15:54:09.266 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 8 + 2 2019-12-21T15:54:09.267 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=10 2019-12-21T15:54:09.267 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:54:09.269 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.4 2019-12-21T15:54:09.289 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2019-12-21T15:54:09.289 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2019-12-21T15:54:09.289 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:54:09.289 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:54:09.290 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:54:09.290 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.4.asok 2019-12-21T15:54:09.290 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:54:09.290 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.4.asok dump_scrub_reservations 2019-12-21T15:54:09.357 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:54:09.357 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 0, 2019-12-21T15:54:09.358 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 4, 2019-12-21T15:54:09.358 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:54:09.358 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:54:09.367 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:54:09.367 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.4 2019-12-21T15:54:09.368 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2019-12-21T15:54:09.368 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2019-12-21T15:54:09.368 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:54:09.368 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:54:09.368 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:54:09.368 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.4.asok 2019-12-21T15:54:09.368 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:54:09.369 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.4.asok dump_scrub_reservations 2019-12-21T15:54:09.461 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=4 2019-12-21T15:54:09.461 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2019-12-21T15:54:09.461 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 10 + 4 2019-12-21T15:54:09.462 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=14 2019-12-21T15:54:09.462 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:54:09.464 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.5 2019-12-21T15:54:09.464 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2019-12-21T15:54:09.464 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2019-12-21T15:54:09.464 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:54:09.465 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:54:09.465 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:54:09.465 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.5.asok 2019-12-21T15:54:09.465 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:54:09.465 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.5.asok dump_scrub_reservations 2019-12-21T15:54:09.552 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:54:09.552 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 2, 2019-12-21T15:54:09.552 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 2, 2019-12-21T15:54:09.552 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:54:09.553 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:54:09.561 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:54:09.561 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.5 2019-12-21T15:54:09.561 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2019-12-21T15:54:09.561 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2019-12-21T15:54:09.561 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:54:09.562 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:54:09.562 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:54:09.562 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.5.asok 2019-12-21T15:54:09.562 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:54:09.562 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.5.asok dump_scrub_reservations 2019-12-21T15:54:09.655 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=4 2019-12-21T15:54:09.655 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2019-12-21T15:54:09.656 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 14 + 4 2019-12-21T15:54:09.657 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=18 2019-12-21T15:54:09.657 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:133: TEST_recover_unexpected: '[' 18 = 0 ']' 2019-12-21T15:54:09.657 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:136: TEST_recover_unexpected: '[' 0 -gt 3 ']' 2019-12-21T15:54:09.658 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: expr 338 + 18 2019-12-21T15:54:09.658 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: total=356 2019-12-21T15:54:09.659 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: expr 2 '*' 2 2019-12-21T15:54:09.660 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: sleep 4 2019-12-21T15:54:13.661 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:120: TEST_recover_unexpected: true 2019-12-21T15:54:13.662 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:122: TEST_recover_unexpected: pass=0 2019-12-21T15:54:13.663 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: expr 6 - 1 2019-12-21T15:54:13.664 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: seq 0 5 2019-12-21T15:54:13.665 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:54:13.666 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.0 2019-12-21T15:54:13.666 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2019-12-21T15:54:13.666 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2019-12-21T15:54:13.667 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:54:13.667 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:54:13.667 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:54:13.667 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.0.asok 2019-12-21T15:54:13.668 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:54:13.668 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.0.asok dump_scrub_reservations 2019-12-21T15:54:13.764 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:54:13.764 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 2, 2019-12-21T15:54:13.764 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 2, 2019-12-21T15:54:13.764 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:54:13.764 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:54:13.773 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:54:13.774 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.0 2019-12-21T15:54:13.774 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2019-12-21T15:54:13.774 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2019-12-21T15:54:13.774 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:54:13.775 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:54:13.775 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:54:13.775 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.0.asok 2019-12-21T15:54:13.775 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:54:13.775 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.0.asok dump_scrub_reservations 2019-12-21T15:54:13.871 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=4 2019-12-21T15:54:13.871 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2019-12-21T15:54:13.871 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 0 + 4 2019-12-21T15:54:13.872 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=4 2019-12-21T15:54:13.873 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:54:13.874 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.1 2019-12-21T15:54:13.874 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2019-12-21T15:54:13.874 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2019-12-21T15:54:13.874 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:54:13.874 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:54:13.875 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:54:13.875 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.1.asok 2019-12-21T15:54:13.875 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:54:13.875 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.1.asok dump_scrub_reservations 2019-12-21T15:54:13.962 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:54:13.962 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 2, 2019-12-21T15:54:13.962 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 2, 2019-12-21T15:54:13.962 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:54:13.962 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:54:13.970 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:54:13.971 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.1 2019-12-21T15:54:13.971 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2019-12-21T15:54:13.971 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2019-12-21T15:54:13.971 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:54:13.971 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:54:13.971 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:54:13.972 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.1.asok 2019-12-21T15:54:13.972 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:54:13.972 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.1.asok dump_scrub_reservations 2019-12-21T15:54:14.067 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=4 2019-12-21T15:54:14.067 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2019-12-21T15:54:14.067 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 4 + 4 2019-12-21T15:54:14.068 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=8 2019-12-21T15:54:14.068 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:54:14.070 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.2 2019-12-21T15:54:14.070 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2019-12-21T15:54:14.070 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2019-12-21T15:54:14.070 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:54:14.070 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:54:14.071 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:54:14.071 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.2.asok 2019-12-21T15:54:14.071 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:54:14.071 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.2.asok dump_scrub_reservations 2019-12-21T15:54:14.159 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:54:14.159 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 0, 2019-12-21T15:54:14.159 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 1, 2019-12-21T15:54:14.159 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:54:14.159 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:54:14.167 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:54:14.168 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.2 2019-12-21T15:54:14.168 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2019-12-21T15:54:14.169 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2019-12-21T15:54:14.169 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:54:14.169 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:54:14.169 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:54:14.169 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.2.asok 2019-12-21T15:54:14.169 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:54:14.170 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.2.asok dump_scrub_reservations 2019-12-21T15:54:14.263 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=1 2019-12-21T15:54:14.263 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 1 -gt 4 ']' 2019-12-21T15:54:14.264 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 8 + 1 2019-12-21T15:54:14.264 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=9 2019-12-21T15:54:14.265 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:54:14.266 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.3 2019-12-21T15:54:14.266 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2019-12-21T15:54:14.267 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2019-12-21T15:54:14.267 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:54:14.267 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:54:14.267 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:54:14.267 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.3.asok 2019-12-21T15:54:14.267 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:54:14.268 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.3.asok dump_scrub_reservations 2019-12-21T15:54:14.354 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:54:14.354 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 1, 2019-12-21T15:54:14.354 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 1, 2019-12-21T15:54:14.355 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:54:14.355 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:54:14.363 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:54:14.363 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.3 2019-12-21T15:54:14.363 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2019-12-21T15:54:14.363 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2019-12-21T15:54:14.363 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:54:14.363 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:54:14.364 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:54:14.364 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.3.asok 2019-12-21T15:54:14.364 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:54:14.364 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.3.asok dump_scrub_reservations 2019-12-21T15:54:14.459 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=2 2019-12-21T15:54:14.459 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 2 -gt 4 ']' 2019-12-21T15:54:14.459 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 9 + 2 2019-12-21T15:54:14.460 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=11 2019-12-21T15:54:14.460 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:54:14.462 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.4 2019-12-21T15:54:14.462 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2019-12-21T15:54:14.463 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2019-12-21T15:54:14.463 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:54:14.463 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:54:14.463 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:54:14.463 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.4.asok 2019-12-21T15:54:14.463 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:54:14.463 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.4.asok dump_scrub_reservations 2019-12-21T15:54:14.550 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:54:14.551 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 0, 2019-12-21T15:54:14.551 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 4, 2019-12-21T15:54:14.551 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:54:14.551 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:54:14.559 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:54:14.559 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.4 2019-12-21T15:54:14.560 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2019-12-21T15:54:14.560 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2019-12-21T15:54:14.560 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:54:14.560 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:54:14.560 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:54:14.560 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.4.asok 2019-12-21T15:54:14.561 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:54:14.561 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.4.asok dump_scrub_reservations 2019-12-21T15:54:14.656 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=4 2019-12-21T15:54:14.657 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2019-12-21T15:54:14.657 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 11 + 4 2019-12-21T15:54:14.658 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=15 2019-12-21T15:54:14.658 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:54:14.659 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.5 2019-12-21T15:54:14.659 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2019-12-21T15:54:14.659 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2019-12-21T15:54:14.659 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:54:14.660 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:54:14.660 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:54:14.660 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.5.asok 2019-12-21T15:54:14.660 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:54:14.661 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.5.asok dump_scrub_reservations 2019-12-21T15:54:14.747 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:54:14.747 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 1, 2019-12-21T15:54:14.747 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 2, 2019-12-21T15:54:14.747 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:54:14.747 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:54:14.755 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:54:14.756 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.5 2019-12-21T15:54:14.756 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2019-12-21T15:54:14.756 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2019-12-21T15:54:14.756 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:54:14.756 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:54:14.757 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:54:14.757 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.5.asok 2019-12-21T15:54:14.757 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:54:14.757 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.5.asok dump_scrub_reservations 2019-12-21T15:54:14.852 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=3 2019-12-21T15:54:14.852 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2019-12-21T15:54:14.852 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 15 + 3 2019-12-21T15:54:14.853 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=18 2019-12-21T15:54:14.853 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:133: TEST_recover_unexpected: '[' 18 = 0 ']' 2019-12-21T15:54:14.853 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:136: TEST_recover_unexpected: '[' 0 -gt 3 ']' 2019-12-21T15:54:14.854 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: expr 356 + 18 2019-12-21T15:54:14.854 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: total=374 2019-12-21T15:54:14.855 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: expr 2 '*' 2 2019-12-21T15:54:14.855 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: sleep 4 2019-12-21T15:54:18.857 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:120: TEST_recover_unexpected: true 2019-12-21T15:54:18.857 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:122: TEST_recover_unexpected: pass=0 2019-12-21T15:54:18.858 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: expr 6 - 1 2019-12-21T15:54:18.860 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: seq 0 5 2019-12-21T15:54:18.861 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:54:18.862 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.0 2019-12-21T15:54:18.862 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2019-12-21T15:54:18.863 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2019-12-21T15:54:18.863 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:54:18.863 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:54:18.863 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:54:18.863 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.0.asok 2019-12-21T15:54:18.864 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:54:18.864 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.0.asok dump_scrub_reservations 2019-12-21T15:54:18.961 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:54:18.961 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 1, 2019-12-21T15:54:18.961 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 1, 2019-12-21T15:54:18.961 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:54:18.961 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:54:18.970 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:54:18.970 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.0 2019-12-21T15:54:18.970 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2019-12-21T15:54:18.970 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2019-12-21T15:54:18.971 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:54:18.971 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:54:18.971 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:54:18.971 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.0.asok 2019-12-21T15:54:18.971 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:54:18.971 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.0.asok dump_scrub_reservations 2019-12-21T15:54:19.066 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=2 2019-12-21T15:54:19.066 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 2 -gt 4 ']' 2019-12-21T15:54:19.067 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 0 + 2 2019-12-21T15:54:19.068 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=2 2019-12-21T15:54:19.068 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:54:19.069 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.1 2019-12-21T15:54:19.069 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2019-12-21T15:54:19.070 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2019-12-21T15:54:19.070 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:54:19.070 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:54:19.070 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:54:19.070 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.1.asok 2019-12-21T15:54:19.070 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:54:19.071 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.1.asok dump_scrub_reservations 2019-12-21T15:54:19.159 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:54:19.159 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 2, 2019-12-21T15:54:19.160 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 2, 2019-12-21T15:54:19.160 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:54:19.160 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:54:19.168 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:54:19.168 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.1 2019-12-21T15:54:19.168 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2019-12-21T15:54:19.168 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2019-12-21T15:54:19.169 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:54:19.169 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:54:19.169 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:54:19.169 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.1.asok 2019-12-21T15:54:19.170 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:54:19.170 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.1.asok dump_scrub_reservations 2019-12-21T15:54:19.265 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=4 2019-12-21T15:54:19.265 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2019-12-21T15:54:19.266 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 2 + 4 2019-12-21T15:54:19.266 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=6 2019-12-21T15:54:19.267 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:54:19.268 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.2 2019-12-21T15:54:19.268 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2019-12-21T15:54:19.268 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2019-12-21T15:54:19.268 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:54:19.268 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:54:19.269 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:54:19.269 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.2.asok 2019-12-21T15:54:19.269 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:54:19.269 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.2.asok dump_scrub_reservations 2019-12-21T15:54:19.357 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:54:19.357 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 0, 2019-12-21T15:54:19.357 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 1, 2019-12-21T15:54:19.357 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:54:19.357 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:54:19.365 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:54:19.366 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.2 2019-12-21T15:54:19.366 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2019-12-21T15:54:19.366 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2019-12-21T15:54:19.366 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:54:19.367 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:54:19.367 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:54:19.367 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.2.asok 2019-12-21T15:54:19.367 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:54:19.367 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.2.asok dump_scrub_reservations 2019-12-21T15:54:19.461 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=1 2019-12-21T15:54:19.461 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 1 -gt 4 ']' 2019-12-21T15:54:19.462 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 6 + 1 2019-12-21T15:54:19.462 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=7 2019-12-21T15:54:19.463 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:54:19.464 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.3 2019-12-21T15:54:19.464 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2019-12-21T15:54:19.464 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2019-12-21T15:54:19.465 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:54:19.465 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:54:19.465 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:54:19.465 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.3.asok 2019-12-21T15:54:19.465 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:54:19.465 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.3.asok dump_scrub_reservations 2019-12-21T15:54:19.552 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:54:19.553 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 1, 2019-12-21T15:54:19.553 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 0, 2019-12-21T15:54:19.553 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:54:19.553 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:54:19.561 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:54:19.561 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.3 2019-12-21T15:54:19.561 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2019-12-21T15:54:19.562 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2019-12-21T15:54:19.562 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:54:19.562 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:54:19.562 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:54:19.562 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.3.asok 2019-12-21T15:54:19.563 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:54:19.563 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.3.asok dump_scrub_reservations 2019-12-21T15:54:19.657 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=1 2019-12-21T15:54:19.658 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 1 -gt 4 ']' 2019-12-21T15:54:19.658 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 7 + 1 2019-12-21T15:54:19.659 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=8 2019-12-21T15:54:19.659 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:54:19.660 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.4 2019-12-21T15:54:19.660 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2019-12-21T15:54:19.660 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2019-12-21T15:54:19.661 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:54:19.661 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:54:19.661 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:54:19.661 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.4.asok 2019-12-21T15:54:19.662 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:54:19.662 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.4.asok dump_scrub_reservations 2019-12-21T15:54:19.748 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:54:19.749 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 0, 2019-12-21T15:54:19.749 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 3, 2019-12-21T15:54:19.749 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:54:19.749 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:54:19.758 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:54:19.758 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.4 2019-12-21T15:54:19.758 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2019-12-21T15:54:19.758 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2019-12-21T15:54:19.758 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:54:19.759 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:54:19.759 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:54:19.759 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.4.asok 2019-12-21T15:54:19.759 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:54:19.759 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.4.asok dump_scrub_reservations 2019-12-21T15:54:19.854 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=3 2019-12-21T15:54:19.854 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2019-12-21T15:54:19.855 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 8 + 3 2019-12-21T15:54:19.855 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=11 2019-12-21T15:54:19.855 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:54:19.857 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.5 2019-12-21T15:54:19.857 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2019-12-21T15:54:19.858 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2019-12-21T15:54:19.858 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:54:19.858 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:54:19.858 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:54:19.858 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.5.asok 2019-12-21T15:54:19.858 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:54:19.859 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.5.asok dump_scrub_reservations 2019-12-21T15:54:19.944 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:54:19.944 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 0, 2019-12-21T15:54:19.945 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 1, 2019-12-21T15:54:19.945 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:54:19.945 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:54:19.953 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:54:19.954 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.5 2019-12-21T15:54:19.954 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2019-12-21T15:54:19.954 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2019-12-21T15:54:19.954 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:54:19.954 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:54:19.954 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:54:19.955 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.5.asok 2019-12-21T15:54:19.955 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:54:19.955 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.5.asok dump_scrub_reservations 2019-12-21T15:54:20.050 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=1 2019-12-21T15:54:20.050 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 1 -gt 4 ']' 2019-12-21T15:54:20.050 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 11 + 1 2019-12-21T15:54:20.051 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=12 2019-12-21T15:54:20.051 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:133: TEST_recover_unexpected: '[' 12 = 0 ']' 2019-12-21T15:54:20.051 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:136: TEST_recover_unexpected: '[' 0 -gt 3 ']' 2019-12-21T15:54:20.052 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: expr 374 + 12 2019-12-21T15:54:20.052 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: total=386 2019-12-21T15:54:20.053 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: expr 2 '*' 2 2019-12-21T15:54:20.054 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: sleep 4 2019-12-21T15:54:24.055 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:120: TEST_recover_unexpected: true 2019-12-21T15:54:24.055 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:122: TEST_recover_unexpected: pass=0 2019-12-21T15:54:24.056 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: expr 6 - 1 2019-12-21T15:54:24.058 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: seq 0 5 2019-12-21T15:54:24.059 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:54:24.060 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.0 2019-12-21T15:54:24.060 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2019-12-21T15:54:24.060 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2019-12-21T15:54:24.061 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:54:24.061 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:54:24.061 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:54:24.061 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.0.asok 2019-12-21T15:54:24.062 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:54:24.062 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.0.asok dump_scrub_reservations 2019-12-21T15:54:24.158 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:54:24.159 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 1, 2019-12-21T15:54:24.159 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 2, 2019-12-21T15:54:24.159 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:54:24.159 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:54:24.168 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:54:24.168 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.0 2019-12-21T15:54:24.168 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2019-12-21T15:54:24.169 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2019-12-21T15:54:24.169 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:54:24.169 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:54:24.169 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:54:24.169 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.0.asok 2019-12-21T15:54:24.169 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:54:24.169 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.0.asok dump_scrub_reservations 2019-12-21T15:54:24.265 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=3 2019-12-21T15:54:24.266 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2019-12-21T15:54:24.266 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 0 + 3 2019-12-21T15:54:24.267 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=3 2019-12-21T15:54:24.267 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:54:24.268 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.1 2019-12-21T15:54:24.268 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2019-12-21T15:54:24.269 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2019-12-21T15:54:24.269 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:54:24.269 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:54:24.269 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:54:24.269 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.1.asok 2019-12-21T15:54:24.269 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:54:24.270 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.1.asok dump_scrub_reservations 2019-12-21T15:54:24.356 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:54:24.356 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 2, 2019-12-21T15:54:24.356 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 2, 2019-12-21T15:54:24.357 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:54:24.357 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:54:24.365 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:54:24.365 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.1 2019-12-21T15:54:24.365 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2019-12-21T15:54:24.365 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2019-12-21T15:54:24.365 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:54:24.366 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:54:24.366 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:54:24.366 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.1.asok 2019-12-21T15:54:24.366 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:54:24.366 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.1.asok dump_scrub_reservations 2019-12-21T15:54:24.460 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=4 2019-12-21T15:54:24.460 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2019-12-21T15:54:24.461 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 3 + 4 2019-12-21T15:54:24.461 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=7 2019-12-21T15:54:24.461 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:54:24.463 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.2 2019-12-21T15:54:24.463 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2019-12-21T15:54:24.463 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2019-12-21T15:54:24.463 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:54:24.463 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:54:24.464 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:54:24.464 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.2.asok 2019-12-21T15:54:24.464 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:54:24.464 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.2.asok dump_scrub_reservations 2019-12-21T15:54:24.551 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:54:24.551 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 0, 2019-12-21T15:54:24.551 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 2, 2019-12-21T15:54:24.551 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:54:24.552 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:54:24.559 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:54:24.560 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.2 2019-12-21T15:54:24.560 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2019-12-21T15:54:24.560 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2019-12-21T15:54:24.560 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:54:24.560 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:54:24.560 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:54:24.560 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.2.asok 2019-12-21T15:54:24.561 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:54:24.561 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.2.asok dump_scrub_reservations 2019-12-21T15:54:24.656 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=2 2019-12-21T15:54:24.657 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 2 -gt 4 ']' 2019-12-21T15:54:24.657 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 7 + 2 2019-12-21T15:54:24.658 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=9 2019-12-21T15:54:24.658 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:54:24.659 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.3 2019-12-21T15:54:24.659 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2019-12-21T15:54:24.660 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2019-12-21T15:54:24.660 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:54:24.660 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:54:24.660 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:54:24.660 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.3.asok 2019-12-21T15:54:24.660 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:54:24.661 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.3.asok dump_scrub_reservations 2019-12-21T15:54:24.748 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:54:24.748 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 1, 2019-12-21T15:54:24.749 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 0, 2019-12-21T15:54:24.749 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:54:24.749 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:54:24.757 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:54:24.757 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.3 2019-12-21T15:54:24.757 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2019-12-21T15:54:24.758 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2019-12-21T15:54:24.758 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:54:24.758 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:54:24.758 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:54:24.758 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.3.asok 2019-12-21T15:54:24.758 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:54:24.758 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.3.asok dump_scrub_reservations 2019-12-21T15:54:24.852 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=1 2019-12-21T15:54:24.853 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 1 -gt 4 ']' 2019-12-21T15:54:24.853 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 9 + 1 2019-12-21T15:54:24.854 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=10 2019-12-21T15:54:24.855 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:54:24.856 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.4 2019-12-21T15:54:24.856 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2019-12-21T15:54:24.856 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2019-12-21T15:54:24.856 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:54:24.856 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:54:24.856 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:54:24.857 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.4.asok 2019-12-21T15:54:24.857 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:54:24.857 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.4.asok dump_scrub_reservations 2019-12-21T15:54:24.945 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:54:24.945 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 1, 2019-12-21T15:54:24.945 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 3, 2019-12-21T15:54:24.945 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:54:24.945 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:54:24.954 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:54:24.954 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.4 2019-12-21T15:54:24.954 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2019-12-21T15:54:24.954 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2019-12-21T15:54:24.954 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:54:24.955 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:54:24.955 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:54:24.955 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.4.asok 2019-12-21T15:54:24.955 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:54:24.955 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.4.asok dump_scrub_reservations 2019-12-21T15:54:25.049 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=4 2019-12-21T15:54:25.049 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2019-12-21T15:54:25.050 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 10 + 4 2019-12-21T15:54:25.051 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=14 2019-12-21T15:54:25.051 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:54:25.052 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.5 2019-12-21T15:54:25.052 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2019-12-21T15:54:25.053 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2019-12-21T15:54:25.053 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:54:25.053 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:54:25.053 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:54:25.053 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.5.asok 2019-12-21T15:54:25.053 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:54:25.055 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.5.asok dump_scrub_reservations 2019-12-21T15:54:25.141 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:54:25.141 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 0, 2019-12-21T15:54:25.141 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 1, 2019-12-21T15:54:25.141 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:54:25.142 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:54:25.149 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:54:25.150 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.5 2019-12-21T15:54:25.150 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2019-12-21T15:54:25.150 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2019-12-21T15:54:25.150 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:54:25.150 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:54:25.150 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:54:25.150 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.5.asok 2019-12-21T15:54:25.151 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:54:25.151 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.5.asok dump_scrub_reservations 2019-12-21T15:54:25.245 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=1 2019-12-21T15:54:25.245 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 1 -gt 4 ']' 2019-12-21T15:54:25.246 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 14 + 1 2019-12-21T15:54:25.247 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=15 2019-12-21T15:54:25.247 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:133: TEST_recover_unexpected: '[' 15 = 0 ']' 2019-12-21T15:54:25.247 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:136: TEST_recover_unexpected: '[' 0 -gt 3 ']' 2019-12-21T15:54:25.248 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: expr 386 + 15 2019-12-21T15:54:25.248 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: total=401 2019-12-21T15:54:25.249 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: expr 2 '*' 2 2019-12-21T15:54:25.250 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: sleep 4 2019-12-21T15:54:29.252 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:120: TEST_recover_unexpected: true 2019-12-21T15:54:29.252 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:122: TEST_recover_unexpected: pass=0 2019-12-21T15:54:29.253 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: expr 6 - 1 2019-12-21T15:54:29.254 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: seq 0 5 2019-12-21T15:54:29.255 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:54:29.256 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.0 2019-12-21T15:54:29.257 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2019-12-21T15:54:29.257 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2019-12-21T15:54:29.257 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:54:29.257 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:54:29.257 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:54:29.257 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.0.asok 2019-12-21T15:54:29.258 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:54:29.258 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.0.asok dump_scrub_reservations 2019-12-21T15:54:29.355 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:54:29.355 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 1, 2019-12-21T15:54:29.355 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 2, 2019-12-21T15:54:29.355 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:54:29.355 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:54:29.364 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:54:29.364 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.0 2019-12-21T15:54:29.364 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2019-12-21T15:54:29.364 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2019-12-21T15:54:29.364 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:54:29.365 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:54:29.365 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:54:29.365 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.0.asok 2019-12-21T15:54:29.365 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:54:29.366 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.0.asok dump_scrub_reservations 2019-12-21T15:54:29.461 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=3 2019-12-21T15:54:29.462 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2019-12-21T15:54:29.462 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 0 + 3 2019-12-21T15:54:29.463 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=3 2019-12-21T15:54:29.463 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:54:29.464 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.1 2019-12-21T15:54:29.464 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2019-12-21T15:54:29.464 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2019-12-21T15:54:29.465 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:54:29.465 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:54:29.465 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:54:29.465 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.1.asok 2019-12-21T15:54:29.466 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:54:29.466 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.1.asok dump_scrub_reservations 2019-12-21T15:54:29.553 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:54:29.553 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 2, 2019-12-21T15:54:29.553 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 2, 2019-12-21T15:54:29.554 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:54:29.554 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:54:29.562 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:54:29.562 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.1 2019-12-21T15:54:29.562 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2019-12-21T15:54:29.562 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2019-12-21T15:54:29.562 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:54:29.563 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:54:29.563 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:54:29.563 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.1.asok 2019-12-21T15:54:29.563 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:54:29.563 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.1.asok dump_scrub_reservations 2019-12-21T15:54:29.658 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=4 2019-12-21T15:54:29.658 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2019-12-21T15:54:29.658 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 3 + 4 2019-12-21T15:54:29.659 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=7 2019-12-21T15:54:29.659 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:54:29.660 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.2 2019-12-21T15:54:29.661 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2019-12-21T15:54:29.661 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2019-12-21T15:54:29.661 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:54:29.661 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:54:29.661 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:54:29.661 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.2.asok 2019-12-21T15:54:29.661 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:54:29.662 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.2.asok dump_scrub_reservations 2019-12-21T15:54:29.749 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:54:29.749 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 0, 2019-12-21T15:54:29.749 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 2, 2019-12-21T15:54:29.750 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:54:29.750 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:54:29.757 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:54:29.758 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.2 2019-12-21T15:54:29.758 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2019-12-21T15:54:29.758 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2019-12-21T15:54:29.758 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:54:29.758 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:54:29.758 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:54:29.759 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.2.asok 2019-12-21T15:54:29.759 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:54:29.759 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.2.asok dump_scrub_reservations 2019-12-21T15:54:29.854 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=2 2019-12-21T15:54:29.854 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 2 -gt 4 ']' 2019-12-21T15:54:29.855 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 7 + 2 2019-12-21T15:54:29.855 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=9 2019-12-21T15:54:29.856 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:54:29.857 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.3 2019-12-21T15:54:29.857 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2019-12-21T15:54:29.857 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2019-12-21T15:54:29.857 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:54:29.857 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:54:29.858 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:54:29.858 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.3.asok 2019-12-21T15:54:29.858 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:54:29.858 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.3.asok dump_scrub_reservations 2019-12-21T15:54:29.947 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:54:29.947 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 1, 2019-12-21T15:54:29.947 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 0, 2019-12-21T15:54:29.947 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:54:29.948 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:54:29.956 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:54:29.956 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.3 2019-12-21T15:54:29.956 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2019-12-21T15:54:29.957 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2019-12-21T15:54:29.957 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:54:29.957 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:54:29.957 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:54:29.957 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.3.asok 2019-12-21T15:54:29.957 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:54:29.957 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.3.asok dump_scrub_reservations 2019-12-21T15:54:30.052 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=1 2019-12-21T15:54:30.052 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 1 -gt 4 ']' 2019-12-21T15:54:30.053 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 9 + 1 2019-12-21T15:54:30.054 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=10 2019-12-21T15:54:30.054 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:54:30.055 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.4 2019-12-21T15:54:30.056 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2019-12-21T15:54:30.056 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2019-12-21T15:54:30.056 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:54:30.056 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:54:30.056 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:54:30.056 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.4.asok 2019-12-21T15:54:30.056 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:54:30.057 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.4.asok dump_scrub_reservations 2019-12-21T15:54:30.143 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:54:30.144 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 1, 2019-12-21T15:54:30.144 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 3, 2019-12-21T15:54:30.144 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:54:30.144 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:54:30.153 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:54:30.153 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.4 2019-12-21T15:54:30.154 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2019-12-21T15:54:30.154 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2019-12-21T15:54:30.154 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:54:30.154 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:54:30.155 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:54:30.155 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.4.asok 2019-12-21T15:54:30.155 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:54:30.155 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.4.asok dump_scrub_reservations 2019-12-21T15:54:30.249 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=4 2019-12-21T15:54:30.249 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2019-12-21T15:54:30.249 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 10 + 4 2019-12-21T15:54:30.251 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=14 2019-12-21T15:54:30.251 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:54:30.252 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.5 2019-12-21T15:54:30.252 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2019-12-21T15:54:30.252 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2019-12-21T15:54:30.252 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:54:30.253 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:54:30.253 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:54:30.253 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.5.asok 2019-12-21T15:54:30.253 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:54:30.253 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.5.asok dump_scrub_reservations 2019-12-21T15:54:30.340 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:54:30.340 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 0, 2019-12-21T15:54:30.341 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 1, 2019-12-21T15:54:30.341 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:54:30.341 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:54:30.349 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:54:30.350 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.5 2019-12-21T15:54:30.350 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2019-12-21T15:54:30.350 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2019-12-21T15:54:30.350 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:54:30.350 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:54:30.350 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:54:30.350 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.5.asok 2019-12-21T15:54:30.351 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:54:30.351 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.5.asok dump_scrub_reservations 2019-12-21T15:54:30.445 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=1 2019-12-21T15:54:30.445 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 1 -gt 4 ']' 2019-12-21T15:54:30.445 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 14 + 1 2019-12-21T15:54:30.446 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=15 2019-12-21T15:54:30.446 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:133: TEST_recover_unexpected: '[' 15 = 0 ']' 2019-12-21T15:54:30.446 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:136: TEST_recover_unexpected: '[' 0 -gt 3 ']' 2019-12-21T15:54:30.447 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: expr 401 + 15 2019-12-21T15:54:30.447 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: total=416 2019-12-21T15:54:30.448 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: expr 2 '*' 2 2019-12-21T15:54:30.449 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: sleep 4 2019-12-21T15:54:34.451 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:120: TEST_recover_unexpected: true 2019-12-21T15:54:34.451 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:122: TEST_recover_unexpected: pass=0 2019-12-21T15:54:34.452 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: expr 6 - 1 2019-12-21T15:54:34.453 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: seq 0 5 2019-12-21T15:54:34.456 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:54:34.456 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.0 2019-12-21T15:54:34.456 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2019-12-21T15:54:34.457 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2019-12-21T15:54:34.457 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:54:34.457 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:54:34.457 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:54:34.457 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.0.asok 2019-12-21T15:54:34.457 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:54:34.457 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.0.asok dump_scrub_reservations 2019-12-21T15:54:34.553 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:54:34.554 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 1, 2019-12-21T15:54:34.554 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 2, 2019-12-21T15:54:34.554 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:54:34.554 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:54:34.563 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:54:34.563 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.0 2019-12-21T15:54:34.563 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2019-12-21T15:54:34.563 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2019-12-21T15:54:34.563 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:54:34.564 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:54:34.564 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:54:34.564 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.0.asok 2019-12-21T15:54:34.564 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:54:34.564 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.0.asok dump_scrub_reservations 2019-12-21T15:54:34.660 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=3 2019-12-21T15:54:34.660 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2019-12-21T15:54:34.660 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 0 + 3 2019-12-21T15:54:34.661 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=3 2019-12-21T15:54:34.661 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:54:34.663 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.1 2019-12-21T15:54:34.663 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2019-12-21T15:54:34.663 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2019-12-21T15:54:34.663 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:54:34.663 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:54:34.663 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:54:34.664 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.1.asok 2019-12-21T15:54:34.664 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:54:34.664 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.1.asok dump_scrub_reservations 2019-12-21T15:54:34.752 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:54:34.752 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 2, 2019-12-21T15:54:34.752 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 2, 2019-12-21T15:54:34.752 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:54:34.752 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:54:34.761 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:54:34.761 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.1 2019-12-21T15:54:34.761 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2019-12-21T15:54:34.761 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2019-12-21T15:54:34.761 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:54:34.762 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:54:34.762 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:54:34.762 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.1.asok 2019-12-21T15:54:34.762 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:54:34.762 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.1.asok dump_scrub_reservations 2019-12-21T15:54:34.856 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=4 2019-12-21T15:54:34.857 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2019-12-21T15:54:34.857 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 3 + 4 2019-12-21T15:54:34.858 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=7 2019-12-21T15:54:34.858 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:54:34.859 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.2 2019-12-21T15:54:34.860 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2019-12-21T15:54:34.860 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2019-12-21T15:54:34.860 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:54:34.860 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:54:34.860 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:54:34.860 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.2.asok 2019-12-21T15:54:34.861 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:54:34.861 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.2.asok dump_scrub_reservations 2019-12-21T15:54:34.948 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:54:34.949 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 0, 2019-12-21T15:54:34.949 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 2, 2019-12-21T15:54:34.949 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:54:34.949 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:54:34.957 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:54:34.957 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.2 2019-12-21T15:54:34.957 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2019-12-21T15:54:34.958 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2019-12-21T15:54:34.958 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:54:34.958 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:54:34.958 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:54:34.958 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.2.asok 2019-12-21T15:54:34.958 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:54:34.959 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.2.asok dump_scrub_reservations 2019-12-21T15:54:35.053 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=2 2019-12-21T15:54:35.053 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 2 -gt 4 ']' 2019-12-21T15:54:35.054 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 7 + 2 2019-12-21T15:54:35.057 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=9 2019-12-21T15:54:35.057 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:54:35.057 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.3 2019-12-21T15:54:35.057 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2019-12-21T15:54:35.058 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2019-12-21T15:54:35.058 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:54:35.058 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:54:35.058 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:54:35.058 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.3.asok 2019-12-21T15:54:35.058 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:54:35.058 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.3.asok dump_scrub_reservations 2019-12-21T15:54:35.145 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:54:35.145 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 1, 2019-12-21T15:54:35.145 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 0, 2019-12-21T15:54:35.145 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:54:35.146 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:54:35.154 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:54:35.154 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.3 2019-12-21T15:54:35.154 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2019-12-21T15:54:35.154 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2019-12-21T15:54:35.154 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:54:35.155 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:54:35.155 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:54:35.155 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.3.asok 2019-12-21T15:54:35.155 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:54:35.155 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.3.asok dump_scrub_reservations 2019-12-21T15:54:35.251 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=1 2019-12-21T15:54:35.252 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 1 -gt 4 ']' 2019-12-21T15:54:35.253 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 9 + 1 2019-12-21T15:54:35.253 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=10 2019-12-21T15:54:35.253 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:54:35.254 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.4 2019-12-21T15:54:35.254 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2019-12-21T15:54:35.254 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2019-12-21T15:54:35.255 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:54:35.255 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:54:35.255 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:54:35.255 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.4.asok 2019-12-21T15:54:35.255 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:54:35.255 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.4.asok dump_scrub_reservations 2019-12-21T15:54:35.340 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:54:35.340 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 1, 2019-12-21T15:54:35.341 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 3, 2019-12-21T15:54:35.341 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:54:35.341 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:54:35.349 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:54:35.349 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.4 2019-12-21T15:54:35.350 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2019-12-21T15:54:35.350 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2019-12-21T15:54:35.351 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:54:35.351 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:54:35.351 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:54:35.351 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.4.asok 2019-12-21T15:54:35.351 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:54:35.351 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.4.asok dump_scrub_reservations 2019-12-21T15:54:35.444 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=4 2019-12-21T15:54:35.444 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2019-12-21T15:54:35.445 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 10 + 4 2019-12-21T15:54:35.446 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=14 2019-12-21T15:54:35.446 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:54:35.447 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.5 2019-12-21T15:54:35.447 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2019-12-21T15:54:35.448 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2019-12-21T15:54:35.448 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:54:35.448 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:54:35.448 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:54:35.448 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.5.asok 2019-12-21T15:54:35.448 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:54:35.449 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.5.asok dump_scrub_reservations 2019-12-21T15:54:35.536 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:54:35.536 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 0, 2019-12-21T15:54:35.536 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 1, 2019-12-21T15:54:35.536 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:54:35.536 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:54:35.544 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:54:35.545 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.5 2019-12-21T15:54:35.545 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2019-12-21T15:54:35.545 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2019-12-21T15:54:35.545 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:54:35.545 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:54:35.545 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:54:35.546 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.5.asok 2019-12-21T15:54:35.546 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:54:35.546 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.5.asok dump_scrub_reservations 2019-12-21T15:54:35.641 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=1 2019-12-21T15:54:35.641 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 1 -gt 4 ']' 2019-12-21T15:54:35.641 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 14 + 1 2019-12-21T15:54:35.642 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=15 2019-12-21T15:54:35.642 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:133: TEST_recover_unexpected: '[' 15 = 0 ']' 2019-12-21T15:54:35.642 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:136: TEST_recover_unexpected: '[' 0 -gt 3 ']' 2019-12-21T15:54:35.643 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: expr 416 + 15 2019-12-21T15:54:35.643 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: total=431 2019-12-21T15:54:35.644 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: expr 2 '*' 2 2019-12-21T15:54:35.645 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: sleep 4 2019-12-21T15:54:39.646 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:120: TEST_recover_unexpected: true 2019-12-21T15:54:39.647 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:122: TEST_recover_unexpected: pass=0 2019-12-21T15:54:39.648 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: expr 6 - 1 2019-12-21T15:54:39.649 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: seq 0 5 2019-12-21T15:54:39.650 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:54:39.651 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.0 2019-12-21T15:54:39.651 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2019-12-21T15:54:39.651 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2019-12-21T15:54:39.652 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:54:39.652 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:54:39.652 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:54:39.653 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.0.asok 2019-12-21T15:54:39.653 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:54:39.653 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.0.asok dump_scrub_reservations 2019-12-21T15:54:39.748 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:54:39.749 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 0, 2019-12-21T15:54:39.749 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 2, 2019-12-21T15:54:39.749 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:54:39.749 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:54:39.758 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:54:39.758 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.0 2019-12-21T15:54:39.759 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2019-12-21T15:54:39.759 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2019-12-21T15:54:39.759 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:54:39.759 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:54:39.759 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:54:39.759 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.0.asok 2019-12-21T15:54:39.760 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:54:39.760 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.0.asok dump_scrub_reservations 2019-12-21T15:54:39.854 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=3 2019-12-21T15:54:39.855 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2019-12-21T15:54:39.855 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 0 + 3 2019-12-21T15:54:39.856 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=3 2019-12-21T15:54:39.856 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:54:39.857 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.1 2019-12-21T15:54:39.858 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2019-12-21T15:54:39.858 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2019-12-21T15:54:39.858 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:54:39.858 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:54:39.858 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:54:39.859 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.1.asok 2019-12-21T15:54:39.859 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:54:39.859 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.1.asok dump_scrub_reservations 2019-12-21T15:54:39.945 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:54:39.945 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 2, 2019-12-21T15:54:39.945 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 2, 2019-12-21T15:54:39.946 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:54:39.946 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:54:39.954 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:54:39.954 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.1 2019-12-21T15:54:39.954 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2019-12-21T15:54:39.954 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2019-12-21T15:54:39.955 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:54:39.955 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:54:39.955 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:54:39.955 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.1.asok 2019-12-21T15:54:39.955 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:54:39.956 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.1.asok dump_scrub_reservations 2019-12-21T15:54:40.051 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=4 2019-12-21T15:54:40.051 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2019-12-21T15:54:40.052 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 3 + 4 2019-12-21T15:54:40.053 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=7 2019-12-21T15:54:40.054 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:54:40.054 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.2 2019-12-21T15:54:40.054 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2019-12-21T15:54:40.054 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2019-12-21T15:54:40.054 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:54:40.054 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:54:40.054 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:54:40.055 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.2.asok 2019-12-21T15:54:40.055 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:54:40.055 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.2.asok dump_scrub_reservations 2019-12-21T15:54:40.142 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:54:40.142 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 0, 2019-12-21T15:54:40.143 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 2, 2019-12-21T15:54:40.143 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:54:40.143 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:54:40.152 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:54:40.152 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.2 2019-12-21T15:54:40.152 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2019-12-21T15:54:40.152 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2019-12-21T15:54:40.153 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:54:40.153 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:54:40.153 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:54:40.153 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.2.asok 2019-12-21T15:54:40.153 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:54:40.153 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.2.asok dump_scrub_reservations 2019-12-21T15:54:40.247 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=2 2019-12-21T15:54:40.247 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 2 -gt 4 ']' 2019-12-21T15:54:40.248 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 7 + 2 2019-12-21T15:54:40.249 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=9 2019-12-21T15:54:40.249 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:54:40.251 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.3 2019-12-21T15:54:40.251 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2019-12-21T15:54:40.251 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2019-12-21T15:54:40.251 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:54:40.252 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:54:40.252 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:54:40.252 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.3.asok 2019-12-21T15:54:40.252 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:54:40.252 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.3.asok dump_scrub_reservations 2019-12-21T15:54:40.339 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:54:40.339 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 1, 2019-12-21T15:54:40.339 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 0, 2019-12-21T15:54:40.339 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:54:40.340 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:54:40.350 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:54:40.350 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.3 2019-12-21T15:54:40.350 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2019-12-21T15:54:40.351 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2019-12-21T15:54:40.351 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:54:40.351 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:54:40.351 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:54:40.352 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.3.asok 2019-12-21T15:54:40.352 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:54:40.352 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.3.asok dump_scrub_reservations 2019-12-21T15:54:40.443 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=1 2019-12-21T15:54:40.444 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 1 -gt 4 ']' 2019-12-21T15:54:40.445 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 9 + 1 2019-12-21T15:54:40.445 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=10 2019-12-21T15:54:40.445 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:54:40.446 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.4 2019-12-21T15:54:40.446 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2019-12-21T15:54:40.447 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2019-12-21T15:54:40.447 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:54:40.447 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:54:40.447 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:54:40.447 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.4.asok 2019-12-21T15:54:40.448 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:54:40.448 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.4.asok dump_scrub_reservations 2019-12-21T15:54:40.533 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:54:40.534 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 1, 2019-12-21T15:54:40.534 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 3, 2019-12-21T15:54:40.534 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:54:40.534 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:54:40.542 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:54:40.543 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.4 2019-12-21T15:54:40.543 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2019-12-21T15:54:40.543 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2019-12-21T15:54:40.543 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:54:40.543 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:54:40.543 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:54:40.544 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.4.asok 2019-12-21T15:54:40.544 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:54:40.544 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.4.asok dump_scrub_reservations 2019-12-21T15:54:40.638 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=4 2019-12-21T15:54:40.639 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2019-12-21T15:54:40.639 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 10 + 4 2019-12-21T15:54:40.640 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=14 2019-12-21T15:54:40.640 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:54:40.650 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.5 2019-12-21T15:54:40.650 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2019-12-21T15:54:40.650 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2019-12-21T15:54:40.651 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:54:40.651 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:54:40.651 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:54:40.651 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.5.asok 2019-12-21T15:54:40.652 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:54:40.652 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.5.asok dump_scrub_reservations 2019-12-21T15:54:40.730 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:54:40.731 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 0, 2019-12-21T15:54:40.731 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 1, 2019-12-21T15:54:40.731 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:54:40.731 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:54:40.739 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:54:40.740 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.5 2019-12-21T15:54:40.740 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2019-12-21T15:54:40.740 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2019-12-21T15:54:40.740 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:54:40.740 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:54:40.740 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:54:40.740 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.5.asok 2019-12-21T15:54:40.741 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:54:40.741 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.5.asok dump_scrub_reservations 2019-12-21T15:54:40.836 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=1 2019-12-21T15:54:40.838 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 1 -gt 4 ']' 2019-12-21T15:54:40.838 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 14 + 1 2019-12-21T15:54:40.839 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=15 2019-12-21T15:54:40.839 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:133: TEST_recover_unexpected: '[' 15 = 0 ']' 2019-12-21T15:54:40.840 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:136: TEST_recover_unexpected: '[' 0 -gt 3 ']' 2019-12-21T15:54:40.842 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: expr 431 + 15 2019-12-21T15:54:40.842 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: total=446 2019-12-21T15:54:40.842 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: expr 2 '*' 2 2019-12-21T15:54:40.842 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: sleep 4 2019-12-21T15:54:44.840 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:120: TEST_recover_unexpected: true 2019-12-21T15:54:44.841 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:122: TEST_recover_unexpected: pass=0 2019-12-21T15:54:44.842 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: expr 6 - 1 2019-12-21T15:54:44.843 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: seq 0 5 2019-12-21T15:54:44.844 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:54:44.845 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.0 2019-12-21T15:54:44.846 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2019-12-21T15:54:44.846 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2019-12-21T15:54:44.846 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:54:44.846 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:54:44.846 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:54:44.846 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.0.asok 2019-12-21T15:54:44.847 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:54:44.847 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.0.asok dump_scrub_reservations 2019-12-21T15:54:44.944 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:54:44.944 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 1, 2019-12-21T15:54:44.944 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 0, 2019-12-21T15:54:44.944 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:54:44.944 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:54:44.953 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:54:44.953 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.0 2019-12-21T15:54:44.953 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2019-12-21T15:54:44.953 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2019-12-21T15:54:44.953 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:54:44.954 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:54:44.954 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:54:44.954 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.0.asok 2019-12-21T15:54:44.954 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:54:44.955 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.0.asok dump_scrub_reservations 2019-12-21T15:54:45.049 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=1 2019-12-21T15:54:45.049 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 1 -gt 4 ']' 2019-12-21T15:54:45.050 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 0 + 1 2019-12-21T15:54:45.050 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=1 2019-12-21T15:54:45.051 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:54:45.052 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.1 2019-12-21T15:54:45.052 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2019-12-21T15:54:45.052 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2019-12-21T15:54:45.053 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:54:45.053 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:54:45.053 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:54:45.053 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.1.asok 2019-12-21T15:54:45.053 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:54:45.053 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.1.asok dump_scrub_reservations 2019-12-21T15:54:45.141 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:54:45.141 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 3, 2019-12-21T15:54:45.141 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 1, 2019-12-21T15:54:45.141 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:54:45.141 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:54:45.150 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:54:45.150 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.1 2019-12-21T15:54:45.150 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2019-12-21T15:54:45.150 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2019-12-21T15:54:45.150 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:54:45.151 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:54:45.151 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:54:45.151 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.1.asok 2019-12-21T15:54:45.151 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:54:45.151 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.1.asok dump_scrub_reservations 2019-12-21T15:54:45.247 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=4 2019-12-21T15:54:45.248 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2019-12-21T15:54:45.248 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 1 + 4 2019-12-21T15:54:45.248 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=5 2019-12-21T15:54:45.248 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:54:45.250 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.2 2019-12-21T15:54:45.250 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2019-12-21T15:54:45.250 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2019-12-21T15:54:45.250 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:54:45.250 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:54:45.250 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:54:45.250 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.2.asok 2019-12-21T15:54:45.251 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:54:45.251 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.2.asok dump_scrub_reservations 2019-12-21T15:54:45.338 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:54:45.338 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 0, 2019-12-21T15:54:45.338 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 3, 2019-12-21T15:54:45.338 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:54:45.338 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:54:45.347 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:54:45.348 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.2 2019-12-21T15:54:45.348 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2019-12-21T15:54:45.348 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2019-12-21T15:54:45.348 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:54:45.348 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:54:45.348 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:54:45.348 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.2.asok 2019-12-21T15:54:45.349 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:54:45.349 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.2.asok dump_scrub_reservations 2019-12-21T15:54:45.445 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=3 2019-12-21T15:54:45.445 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2019-12-21T15:54:45.446 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 5 + 3 2019-12-21T15:54:45.446 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=8 2019-12-21T15:54:45.446 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:54:45.447 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.3 2019-12-21T15:54:45.447 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2019-12-21T15:54:45.447 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2019-12-21T15:54:45.448 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:54:45.448 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:54:45.448 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:54:45.448 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.3.asok 2019-12-21T15:54:45.449 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:54:45.450 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.3.asok dump_scrub_reservations 2019-12-21T15:54:45.535 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:54:45.536 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 0, 2019-12-21T15:54:45.536 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 0, 2019-12-21T15:54:45.536 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:54:45.536 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:54:45.544 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:54:45.544 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.3 2019-12-21T15:54:45.545 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2019-12-21T15:54:45.545 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2019-12-21T15:54:45.545 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:54:45.545 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:54:45.545 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:54:45.545 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.3.asok 2019-12-21T15:54:45.545 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:54:45.546 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.3.asok dump_scrub_reservations 2019-12-21T15:54:45.639 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=0 2019-12-21T15:54:45.640 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 0 -gt 4 ']' 2019-12-21T15:54:45.640 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 8 + 0 2019-12-21T15:54:45.641 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=8 2019-12-21T15:54:45.641 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:54:45.642 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.4 2019-12-21T15:54:45.642 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2019-12-21T15:54:45.642 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2019-12-21T15:54:45.643 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:54:45.643 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:54:45.643 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:54:45.643 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.4.asok 2019-12-21T15:54:45.644 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:54:45.644 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.4.asok dump_scrub_reservations 2019-12-21T15:54:45.731 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:54:45.731 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 0, 2019-12-21T15:54:45.731 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 4, 2019-12-21T15:54:45.731 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:54:45.731 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:54:45.740 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:54:45.740 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.4 2019-12-21T15:54:45.740 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2019-12-21T15:54:45.740 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2019-12-21T15:54:45.740 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:54:45.740 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:54:45.741 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:54:45.741 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.4.asok 2019-12-21T15:54:45.741 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:54:45.741 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.4.asok dump_scrub_reservations 2019-12-21T15:54:45.836 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=4 2019-12-21T15:54:45.836 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2019-12-21T15:54:45.837 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 8 + 4 2019-12-21T15:54:45.837 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=12 2019-12-21T15:54:45.837 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:54:45.839 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.5 2019-12-21T15:54:45.839 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2019-12-21T15:54:45.839 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2019-12-21T15:54:45.839 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:54:45.839 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:54:45.840 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:54:45.840 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.5.asok 2019-12-21T15:54:45.840 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:54:45.840 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.5.asok dump_scrub_reservations 2019-12-21T15:54:45.927 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:54:45.928 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 0, 2019-12-21T15:54:45.928 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 0, 2019-12-21T15:54:45.928 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:54:45.928 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:54:45.936 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:54:45.936 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.5 2019-12-21T15:54:45.937 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2019-12-21T15:54:45.937 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2019-12-21T15:54:45.937 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:54:45.937 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:54:45.937 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:54:45.937 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.5.asok 2019-12-21T15:54:45.937 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:54:45.938 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.5.asok dump_scrub_reservations 2019-12-21T15:54:46.032 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=0 2019-12-21T15:54:46.032 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 0 -gt 4 ']' 2019-12-21T15:54:46.033 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 12 + 0 2019-12-21T15:54:46.034 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=12 2019-12-21T15:54:46.034 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:133: TEST_recover_unexpected: '[' 12 = 0 ']' 2019-12-21T15:54:46.034 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:136: TEST_recover_unexpected: '[' 0 -gt 3 ']' 2019-12-21T15:54:46.034 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: expr 446 + 12 2019-12-21T15:54:46.035 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: total=458 2019-12-21T15:54:46.036 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: expr 2 '*' 2 2019-12-21T15:54:46.036 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: sleep 4 2019-12-21T15:54:50.038 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:120: TEST_recover_unexpected: true 2019-12-21T15:54:50.038 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:122: TEST_recover_unexpected: pass=0 2019-12-21T15:54:50.039 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: expr 6 - 1 2019-12-21T15:54:50.041 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: seq 0 5 2019-12-21T15:54:50.042 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:54:50.043 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.0 2019-12-21T15:54:50.043 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2019-12-21T15:54:50.044 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2019-12-21T15:54:50.044 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:54:50.044 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:54:50.044 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:54:50.044 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.0.asok 2019-12-21T15:54:50.045 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:54:50.045 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.0.asok dump_scrub_reservations 2019-12-21T15:54:50.141 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:54:50.141 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 1, 2019-12-21T15:54:50.142 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 1, 2019-12-21T15:54:50.142 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:54:50.142 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:54:50.150 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:54:50.151 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.0 2019-12-21T15:54:50.151 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2019-12-21T15:54:50.151 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2019-12-21T15:54:50.151 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:54:50.151 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:54:50.151 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:54:50.152 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.0.asok 2019-12-21T15:54:50.152 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:54:50.152 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.0.asok dump_scrub_reservations 2019-12-21T15:54:50.247 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=2 2019-12-21T15:54:50.247 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 2 -gt 4 ']' 2019-12-21T15:54:50.248 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 0 + 2 2019-12-21T15:54:50.248 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=2 2019-12-21T15:54:50.249 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:54:50.250 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.1 2019-12-21T15:54:50.250 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2019-12-21T15:54:50.250 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2019-12-21T15:54:50.251 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:54:50.251 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:54:50.251 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:54:50.251 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.1.asok 2019-12-21T15:54:50.251 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:54:50.251 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.1.asok dump_scrub_reservations 2019-12-21T15:54:50.339 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:54:50.339 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 2, 2019-12-21T15:54:50.339 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 2, 2019-12-21T15:54:50.340 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:54:50.340 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:54:50.349 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:54:50.349 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.1 2019-12-21T15:54:50.349 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2019-12-21T15:54:50.349 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2019-12-21T15:54:50.349 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:54:50.350 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:54:50.350 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:54:50.350 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.1.asok 2019-12-21T15:54:50.350 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:54:50.350 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.1.asok dump_scrub_reservations 2019-12-21T15:54:50.444 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=4 2019-12-21T15:54:50.444 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2019-12-21T15:54:50.444 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 2 + 4 2019-12-21T15:54:50.445 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=6 2019-12-21T15:54:50.445 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:54:50.446 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.2 2019-12-21T15:54:50.447 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2019-12-21T15:54:50.447 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2019-12-21T15:54:50.447 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:54:50.447 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:54:50.447 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:54:50.447 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.2.asok 2019-12-21T15:54:50.448 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:54:50.448 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.2.asok dump_scrub_reservations 2019-12-21T15:54:50.536 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:54:50.536 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 0, 2019-12-21T15:54:50.536 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 2, 2019-12-21T15:54:50.536 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:54:50.536 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:54:50.544 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:54:50.545 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.2 2019-12-21T15:54:50.545 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2019-12-21T15:54:50.545 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2019-12-21T15:54:50.545 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:54:50.545 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:54:50.545 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:54:50.546 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.2.asok 2019-12-21T15:54:50.546 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:54:50.546 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.2.asok dump_scrub_reservations 2019-12-21T15:54:50.640 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=2 2019-12-21T15:54:50.640 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 2 -gt 4 ']' 2019-12-21T15:54:50.641 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 6 + 2 2019-12-21T15:54:50.642 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=8 2019-12-21T15:54:50.642 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:54:50.643 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.3 2019-12-21T15:54:50.643 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2019-12-21T15:54:50.643 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2019-12-21T15:54:50.643 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:54:50.643 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:54:50.644 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:54:50.644 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.3.asok 2019-12-21T15:54:50.644 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:54:50.644 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.3.asok dump_scrub_reservations 2019-12-21T15:54:50.732 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:54:50.732 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 0, 2019-12-21T15:54:50.732 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 0, 2019-12-21T15:54:50.732 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:54:50.733 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:54:50.740 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:54:50.741 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.3 2019-12-21T15:54:50.741 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2019-12-21T15:54:50.741 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2019-12-21T15:54:50.741 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:54:50.741 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:54:50.741 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:54:50.741 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.3.asok 2019-12-21T15:54:50.742 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:54:50.742 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.3.asok dump_scrub_reservations 2019-12-21T15:54:50.838 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=0 2019-12-21T15:54:50.838 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 0 -gt 4 ']' 2019-12-21T15:54:50.838 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 8 + 0 2019-12-21T15:54:50.839 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=8 2019-12-21T15:54:50.839 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:54:50.841 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.4 2019-12-21T15:54:50.841 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2019-12-21T15:54:50.842 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2019-12-21T15:54:50.842 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:54:50.842 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:54:50.842 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:54:50.842 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.4.asok 2019-12-21T15:54:50.842 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:54:50.842 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.4.asok dump_scrub_reservations 2019-12-21T15:54:50.933 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:54:50.933 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 1, 2019-12-21T15:54:50.933 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 3, 2019-12-21T15:54:50.933 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:54:50.933 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:54:50.939 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:54:50.939 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.4 2019-12-21T15:54:50.939 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2019-12-21T15:54:50.939 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2019-12-21T15:54:50.939 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:54:50.940 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:54:50.940 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:54:50.940 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.4.asok 2019-12-21T15:54:50.940 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:54:50.940 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.4.asok dump_scrub_reservations 2019-12-21T15:54:51.035 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=4 2019-12-21T15:54:51.035 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2019-12-21T15:54:51.036 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 8 + 4 2019-12-21T15:54:51.037 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=12 2019-12-21T15:54:51.037 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:54:51.038 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.5 2019-12-21T15:54:51.038 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2019-12-21T15:54:51.039 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2019-12-21T15:54:51.039 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:54:51.039 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:54:51.039 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:54:51.039 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.5.asok 2019-12-21T15:54:51.039 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:54:51.039 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.5.asok dump_scrub_reservations 2019-12-21T15:54:51.126 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:54:51.127 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 0, 2019-12-21T15:54:51.127 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 0, 2019-12-21T15:54:51.127 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:54:51.127 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:54:51.135 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:54:51.135 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.5 2019-12-21T15:54:51.136 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2019-12-21T15:54:51.136 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2019-12-21T15:54:51.136 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:54:51.136 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:54:51.136 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:54:51.136 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.5.asok 2019-12-21T15:54:51.136 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:54:51.137 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.5.asok dump_scrub_reservations 2019-12-21T15:54:51.231 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=0 2019-12-21T15:54:51.231 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 0 -gt 4 ']' 2019-12-21T15:54:51.232 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 12 + 0 2019-12-21T15:54:51.233 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=12 2019-12-21T15:54:51.233 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:133: TEST_recover_unexpected: '[' 12 = 0 ']' 2019-12-21T15:54:51.233 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:136: TEST_recover_unexpected: '[' 0 -gt 3 ']' 2019-12-21T15:54:51.234 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: expr 458 + 12 2019-12-21T15:54:51.234 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: total=470 2019-12-21T15:54:51.235 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: expr 2 '*' 2 2019-12-21T15:54:51.235 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: sleep 4 2019-12-21T15:54:55.237 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:120: TEST_recover_unexpected: true 2019-12-21T15:54:55.238 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:122: TEST_recover_unexpected: pass=0 2019-12-21T15:54:55.239 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: expr 6 - 1 2019-12-21T15:54:55.240 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: seq 0 5 2019-12-21T15:54:55.241 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:54:55.242 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.0 2019-12-21T15:54:55.243 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2019-12-21T15:54:55.243 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2019-12-21T15:54:55.243 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:54:55.243 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:54:55.243 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:54:55.243 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.0.asok 2019-12-21T15:54:55.244 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:54:55.244 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.0.asok dump_scrub_reservations 2019-12-21T15:54:55.343 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:54:55.343 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 1, 2019-12-21T15:54:55.343 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 1, 2019-12-21T15:54:55.343 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:54:55.343 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:54:55.352 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:54:55.352 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.0 2019-12-21T15:54:55.352 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2019-12-21T15:54:55.352 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2019-12-21T15:54:55.353 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:54:55.353 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:54:55.353 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:54:55.353 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.0.asok 2019-12-21T15:54:55.353 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:54:55.354 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.0.asok dump_scrub_reservations 2019-12-21T15:54:55.449 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=2 2019-12-21T15:54:55.449 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 2 -gt 4 ']' 2019-12-21T15:54:55.449 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 0 + 2 2019-12-21T15:54:55.450 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=2 2019-12-21T15:54:55.450 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:54:55.452 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.1 2019-12-21T15:54:55.452 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2019-12-21T15:54:55.452 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2019-12-21T15:54:55.452 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:54:55.452 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:54:55.453 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:54:55.453 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.1.asok 2019-12-21T15:54:55.453 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:54:55.453 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.1.asok dump_scrub_reservations 2019-12-21T15:54:55.542 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:54:55.542 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 1, 2019-12-21T15:54:55.542 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 3, 2019-12-21T15:54:55.543 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:54:55.543 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:54:55.551 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:54:55.551 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.1 2019-12-21T15:54:55.551 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2019-12-21T15:54:55.551 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2019-12-21T15:54:55.551 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:54:55.551 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:54:55.552 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:54:55.552 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.1.asok 2019-12-21T15:54:55.552 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:54:55.552 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.1.asok dump_scrub_reservations 2019-12-21T15:54:55.647 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=4 2019-12-21T15:54:55.647 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2019-12-21T15:54:55.648 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 2 + 4 2019-12-21T15:54:55.649 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=6 2019-12-21T15:54:55.649 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:54:55.650 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.2 2019-12-21T15:54:55.651 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2019-12-21T15:54:55.651 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2019-12-21T15:54:55.651 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:54:55.652 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:54:55.652 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:54:55.652 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.2.asok 2019-12-21T15:54:55.652 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:54:55.652 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.2.asok dump_scrub_reservations 2019-12-21T15:54:55.738 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:54:55.738 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 0, 2019-12-21T15:54:55.738 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 2, 2019-12-21T15:54:55.739 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:54:55.739 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:54:55.747 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:54:55.747 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.2 2019-12-21T15:54:55.747 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2019-12-21T15:54:55.748 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2019-12-21T15:54:55.748 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:54:55.748 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:54:55.748 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:54:55.748 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.2.asok 2019-12-21T15:54:55.748 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:54:55.748 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.2.asok dump_scrub_reservations 2019-12-21T15:54:55.843 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=2 2019-12-21T15:54:55.843 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 2 -gt 4 ']' 2019-12-21T15:54:55.843 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 6 + 2 2019-12-21T15:54:55.845 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=8 2019-12-21T15:54:55.845 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:54:55.846 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.3 2019-12-21T15:54:55.846 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2019-12-21T15:54:55.846 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2019-12-21T15:54:55.846 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:54:55.847 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:54:55.847 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:54:55.847 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.3.asok 2019-12-21T15:54:55.847 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:54:55.847 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.3.asok dump_scrub_reservations 2019-12-21T15:54:55.934 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:54:55.934 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 0, 2019-12-21T15:54:55.934 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 0, 2019-12-21T15:54:55.935 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:54:55.935 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:54:55.943 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:54:55.943 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.3 2019-12-21T15:54:55.943 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2019-12-21T15:54:55.943 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2019-12-21T15:54:55.943 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:54:55.943 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:54:55.944 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:54:55.944 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.3.asok 2019-12-21T15:54:55.944 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:54:55.944 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.3.asok dump_scrub_reservations 2019-12-21T15:54:56.039 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=0 2019-12-21T15:54:56.039 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 0 -gt 4 ']' 2019-12-21T15:54:56.039 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 8 + 0 2019-12-21T15:54:56.040 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=8 2019-12-21T15:54:56.040 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:54:56.042 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.4 2019-12-21T15:54:56.042 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2019-12-21T15:54:56.042 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2019-12-21T15:54:56.042 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:54:56.042 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:54:56.042 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:54:56.042 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.4.asok 2019-12-21T15:54:56.043 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:54:56.043 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.4.asok dump_scrub_reservations 2019-12-21T15:54:56.130 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:54:56.130 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 2, 2019-12-21T15:54:56.130 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 2, 2019-12-21T15:54:56.130 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:54:56.130 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:54:56.139 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:54:56.139 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.4 2019-12-21T15:54:56.139 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2019-12-21T15:54:56.139 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2019-12-21T15:54:56.139 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:54:56.139 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:54:56.140 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:54:56.140 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.4.asok 2019-12-21T15:54:56.140 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:54:56.140 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.4.asok dump_scrub_reservations 2019-12-21T15:54:56.235 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=4 2019-12-21T15:54:56.235 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2019-12-21T15:54:56.235 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 8 + 4 2019-12-21T15:54:56.236 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=12 2019-12-21T15:54:56.236 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:54:56.238 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.5 2019-12-21T15:54:56.238 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2019-12-21T15:54:56.238 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2019-12-21T15:54:56.238 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:54:56.238 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:54:56.238 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:54:56.239 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.5.asok 2019-12-21T15:54:56.239 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:54:56.239 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.5.asok dump_scrub_reservations 2019-12-21T15:54:56.327 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:54:56.327 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 0, 2019-12-21T15:54:56.327 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 0, 2019-12-21T15:54:56.328 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:54:56.328 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:54:56.336 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:54:56.336 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.5 2019-12-21T15:54:56.336 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2019-12-21T15:54:56.336 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2019-12-21T15:54:56.337 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:54:56.337 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:54:56.337 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:54:56.337 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.5.asok 2019-12-21T15:54:56.337 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:54:56.337 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.5.asok dump_scrub_reservations 2019-12-21T15:54:56.432 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=0 2019-12-21T15:54:56.432 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 0 -gt 4 ']' 2019-12-21T15:54:56.432 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 12 + 0 2019-12-21T15:54:56.433 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=12 2019-12-21T15:54:56.433 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:133: TEST_recover_unexpected: '[' 12 = 0 ']' 2019-12-21T15:54:56.433 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:136: TEST_recover_unexpected: '[' 0 -gt 3 ']' 2019-12-21T15:54:56.434 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: expr 470 + 12 2019-12-21T15:54:56.434 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: total=482 2019-12-21T15:54:56.435 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: expr 2 '*' 2 2019-12-21T15:54:56.436 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: sleep 4 2019-12-21T15:55:00.438 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:120: TEST_recover_unexpected: true 2019-12-21T15:55:00.438 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:122: TEST_recover_unexpected: pass=0 2019-12-21T15:55:00.439 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: expr 6 - 1 2019-12-21T15:55:00.440 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: seq 0 5 2019-12-21T15:55:00.441 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:55:00.442 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.0 2019-12-21T15:55:00.442 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2019-12-21T15:55:00.443 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2019-12-21T15:55:00.443 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:55:00.444 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:55:00.444 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:55:00.444 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.0.asok 2019-12-21T15:55:00.444 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:55:00.444 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.0.asok dump_scrub_reservations 2019-12-21T15:55:00.540 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:55:00.540 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 1, 2019-12-21T15:55:00.540 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 1, 2019-12-21T15:55:00.540 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:55:00.540 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:55:00.548 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:55:00.549 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.0 2019-12-21T15:55:00.549 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2019-12-21T15:55:00.549 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2019-12-21T15:55:00.550 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:55:00.550 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:55:00.550 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:55:00.550 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.0.asok 2019-12-21T15:55:00.550 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:55:00.551 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.0.asok dump_scrub_reservations 2019-12-21T15:55:00.645 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=2 2019-12-21T15:55:00.645 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 2 -gt 4 ']' 2019-12-21T15:55:00.646 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 0 + 2 2019-12-21T15:55:00.647 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=2 2019-12-21T15:55:00.647 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:55:00.648 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.1 2019-12-21T15:55:00.648 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2019-12-21T15:55:00.648 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2019-12-21T15:55:00.649 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:55:00.649 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:55:00.649 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:55:00.649 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.1.asok 2019-12-21T15:55:00.650 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:55:00.650 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.1.asok dump_scrub_reservations 2019-12-21T15:55:00.737 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:55:00.737 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 1, 2019-12-21T15:55:00.737 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 3, 2019-12-21T15:55:00.737 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:55:00.737 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:55:00.745 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:55:00.746 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.1 2019-12-21T15:55:00.746 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2019-12-21T15:55:00.746 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2019-12-21T15:55:00.746 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:55:00.746 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:55:00.746 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:55:00.746 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.1.asok 2019-12-21T15:55:00.747 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:55:00.747 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.1.asok dump_scrub_reservations 2019-12-21T15:55:00.842 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=4 2019-12-21T15:55:00.842 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2019-12-21T15:55:00.842 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 2 + 4 2019-12-21T15:55:00.843 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=6 2019-12-21T15:55:00.843 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:55:00.844 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.2 2019-12-21T15:55:00.844 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2019-12-21T15:55:00.845 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2019-12-21T15:55:00.845 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:55:00.845 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:55:00.845 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:55:00.845 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.2.asok 2019-12-21T15:55:00.846 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:55:00.846 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.2.asok dump_scrub_reservations 2019-12-21T15:55:00.934 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:55:00.980 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 0, 2019-12-21T15:55:00.980 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 2, 2019-12-21T15:55:00.980 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:55:00.980 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:55:00.982 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:55:00.982 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.2 2019-12-21T15:55:00.982 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2019-12-21T15:55:00.982 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2019-12-21T15:55:00.982 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:55:00.982 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:55:00.983 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:55:00.983 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.2.asok 2019-12-21T15:55:00.983 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:55:00.983 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.2.asok dump_scrub_reservations 2019-12-21T15:55:01.039 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=2 2019-12-21T15:55:01.039 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 2 -gt 4 ']' 2019-12-21T15:55:01.039 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 6 + 2 2019-12-21T15:55:01.040 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=8 2019-12-21T15:55:01.040 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:55:01.041 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.3 2019-12-21T15:55:01.042 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2019-12-21T15:55:01.042 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2019-12-21T15:55:01.042 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:55:01.042 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:55:01.042 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:55:01.042 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.3.asok 2019-12-21T15:55:01.043 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:55:01.043 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.3.asok dump_scrub_reservations 2019-12-21T15:55:01.131 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:55:01.131 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 0, 2019-12-21T15:55:01.131 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 0, 2019-12-21T15:55:01.132 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:55:01.132 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:55:01.140 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:55:01.140 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.3 2019-12-21T15:55:01.140 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2019-12-21T15:55:01.140 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2019-12-21T15:55:01.140 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:55:01.141 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:55:01.141 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:55:01.141 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.3.asok 2019-12-21T15:55:01.142 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:55:01.143 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.3.asok dump_scrub_reservations 2019-12-21T15:55:01.236 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=0 2019-12-21T15:55:01.236 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 0 -gt 4 ']' 2019-12-21T15:55:01.236 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 8 + 0 2019-12-21T15:55:01.237 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=8 2019-12-21T15:55:01.237 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:55:01.239 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.4 2019-12-21T15:55:01.239 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2019-12-21T15:55:01.239 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2019-12-21T15:55:01.239 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:55:01.239 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:55:01.239 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:55:01.240 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.4.asok 2019-12-21T15:55:01.240 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:55:01.240 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.4.asok dump_scrub_reservations 2019-12-21T15:55:01.328 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:55:01.328 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 2, 2019-12-21T15:55:01.328 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 2, 2019-12-21T15:55:01.328 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:55:01.328 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:55:01.337 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:55:01.337 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.4 2019-12-21T15:55:01.337 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2019-12-21T15:55:01.337 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2019-12-21T15:55:01.337 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:55:01.338 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:55:01.338 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:55:01.338 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.4.asok 2019-12-21T15:55:01.338 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:55:01.338 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.4.asok dump_scrub_reservations 2019-12-21T15:55:01.434 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=4 2019-12-21T15:55:01.434 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2019-12-21T15:55:01.434 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 8 + 4 2019-12-21T15:55:01.434 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=12 2019-12-21T15:55:01.434 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:55:01.435 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.5 2019-12-21T15:55:01.436 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2019-12-21T15:55:01.436 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2019-12-21T15:55:01.436 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:55:01.436 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:55:01.436 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:55:01.436 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.5.asok 2019-12-21T15:55:01.436 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:55:01.437 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.5.asok dump_scrub_reservations 2019-12-21T15:55:01.522 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:55:01.522 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 0, 2019-12-21T15:55:01.523 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 0, 2019-12-21T15:55:01.523 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:55:01.523 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:55:01.531 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:55:01.531 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.5 2019-12-21T15:55:01.531 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2019-12-21T15:55:01.531 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2019-12-21T15:55:01.531 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:55:01.531 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:55:01.532 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:55:01.532 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.5.asok 2019-12-21T15:55:01.532 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:55:01.532 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.5.asok dump_scrub_reservations 2019-12-21T15:55:01.627 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=0 2019-12-21T15:55:01.627 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 0 -gt 4 ']' 2019-12-21T15:55:01.628 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 12 + 0 2019-12-21T15:55:01.629 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=12 2019-12-21T15:55:01.629 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:133: TEST_recover_unexpected: '[' 12 = 0 ']' 2019-12-21T15:55:01.629 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:136: TEST_recover_unexpected: '[' 0 -gt 3 ']' 2019-12-21T15:55:01.630 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: expr 482 + 12 2019-12-21T15:55:01.630 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: total=494 2019-12-21T15:55:01.631 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: expr 2 '*' 2 2019-12-21T15:55:01.632 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: sleep 4 2019-12-21T15:55:05.634 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:120: TEST_recover_unexpected: true 2019-12-21T15:55:05.634 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:122: TEST_recover_unexpected: pass=0 2019-12-21T15:55:05.635 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: expr 6 - 1 2019-12-21T15:55:05.636 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: seq 0 5 2019-12-21T15:55:05.637 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:55:05.638 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.0 2019-12-21T15:55:05.639 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2019-12-21T15:55:05.639 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2019-12-21T15:55:05.639 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:55:05.639 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:55:05.640 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:55:05.640 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.0.asok 2019-12-21T15:55:05.640 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:55:05.641 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.0.asok dump_scrub_reservations 2019-12-21T15:55:05.737 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:55:05.737 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 1, 2019-12-21T15:55:05.738 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 1, 2019-12-21T15:55:05.738 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:55:05.738 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:55:05.746 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:55:05.746 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.0 2019-12-21T15:55:05.747 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2019-12-21T15:55:05.747 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2019-12-21T15:55:05.747 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:55:05.747 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:55:05.747 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:55:05.747 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.0.asok 2019-12-21T15:55:05.747 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:55:05.748 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.0.asok dump_scrub_reservations 2019-12-21T15:55:05.842 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=2 2019-12-21T15:55:05.843 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 2 -gt 4 ']' 2019-12-21T15:55:05.843 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 0 + 2 2019-12-21T15:55:05.844 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=2 2019-12-21T15:55:05.844 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:55:05.846 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.1 2019-12-21T15:55:05.847 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2019-12-21T15:55:05.847 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2019-12-21T15:55:05.847 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:55:05.847 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:55:05.847 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:55:05.847 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.1.asok 2019-12-21T15:55:05.848 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:55:05.848 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.1.asok dump_scrub_reservations 2019-12-21T15:55:05.935 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:55:05.935 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 1, 2019-12-21T15:55:05.935 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 2, 2019-12-21T15:55:05.935 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:55:05.936 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:55:05.944 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:55:05.945 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.1 2019-12-21T15:55:05.945 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2019-12-21T15:55:05.945 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2019-12-21T15:55:05.945 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:55:05.946 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:55:05.946 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:55:05.946 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.1.asok 2019-12-21T15:55:05.946 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:55:05.946 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.1.asok dump_scrub_reservations 2019-12-21T15:55:06.041 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=3 2019-12-21T15:55:06.041 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2019-12-21T15:55:06.042 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 2 + 3 2019-12-21T15:55:06.043 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=5 2019-12-21T15:55:06.043 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:55:06.044 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.2 2019-12-21T15:55:06.045 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2019-12-21T15:55:06.045 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2019-12-21T15:55:06.045 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:55:06.045 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:55:06.045 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:55:06.045 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.2.asok 2019-12-21T15:55:06.046 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:55:06.046 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.2.asok dump_scrub_reservations 2019-12-21T15:55:06.134 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:55:06.134 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 0, 2019-12-21T15:55:06.135 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 3, 2019-12-21T15:55:06.135 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:55:06.135 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:55:06.145 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:55:06.145 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.2 2019-12-21T15:55:06.145 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2019-12-21T15:55:06.146 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2019-12-21T15:55:06.146 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:55:06.146 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:55:06.146 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:55:06.146 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.2.asok 2019-12-21T15:55:06.146 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:55:06.146 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.2.asok dump_scrub_reservations 2019-12-21T15:55:06.238 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=3 2019-12-21T15:55:06.238 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2019-12-21T15:55:06.239 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 5 + 3 2019-12-21T15:55:06.240 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=8 2019-12-21T15:55:06.240 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:55:06.241 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.3 2019-12-21T15:55:06.241 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2019-12-21T15:55:06.241 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2019-12-21T15:55:06.242 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:55:06.242 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:55:06.242 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:55:06.242 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.3.asok 2019-12-21T15:55:06.242 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:55:06.243 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.3.asok dump_scrub_reservations 2019-12-21T15:55:06.331 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:55:06.331 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 1, 2019-12-21T15:55:06.331 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 1, 2019-12-21T15:55:06.331 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:55:06.331 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:55:06.340 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:55:06.341 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.3 2019-12-21T15:55:06.341 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2019-12-21T15:55:06.341 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2019-12-21T15:55:06.341 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:55:06.341 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:55:06.341 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:55:06.342 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.3.asok 2019-12-21T15:55:06.342 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:55:06.342 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.3.asok dump_scrub_reservations 2019-12-21T15:55:06.436 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=2 2019-12-21T15:55:06.436 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 2 -gt 4 ']' 2019-12-21T15:55:06.436 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 8 + 2 2019-12-21T15:55:06.437 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=10 2019-12-21T15:55:06.437 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:55:06.439 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.4 2019-12-21T15:55:06.439 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2019-12-21T15:55:06.439 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2019-12-21T15:55:06.439 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:55:06.440 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:55:06.440 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:55:06.440 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.4.asok 2019-12-21T15:55:06.440 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:55:06.440 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.4.asok dump_scrub_reservations 2019-12-21T15:55:06.527 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:55:06.527 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 3, 2019-12-21T15:55:06.527 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 1, 2019-12-21T15:55:06.527 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:55:06.527 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:55:06.536 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:55:06.536 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.4 2019-12-21T15:55:06.536 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2019-12-21T15:55:06.536 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2019-12-21T15:55:06.536 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:55:06.536 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:55:06.537 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:55:06.537 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.4.asok 2019-12-21T15:55:06.537 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:55:06.537 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.4.asok dump_scrub_reservations 2019-12-21T15:55:06.631 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=4 2019-12-21T15:55:06.631 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2019-12-21T15:55:06.632 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 10 + 4 2019-12-21T15:55:06.633 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=14 2019-12-21T15:55:06.633 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:55:06.636 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.5 2019-12-21T15:55:06.637 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2019-12-21T15:55:06.637 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2019-12-21T15:55:06.637 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:55:06.637 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:55:06.638 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:55:06.638 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.5.asok 2019-12-21T15:55:06.638 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:55:06.638 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.5.asok dump_scrub_reservations 2019-12-21T15:55:06.723 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:55:06.723 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 0, 2019-12-21T15:55:06.723 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 0, 2019-12-21T15:55:06.723 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:55:06.724 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:55:06.732 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:55:06.732 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.5 2019-12-21T15:55:06.732 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2019-12-21T15:55:06.732 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2019-12-21T15:55:06.733 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:55:06.733 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:55:06.733 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:55:06.733 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.5.asok 2019-12-21T15:55:06.733 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:55:06.733 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.5.asok dump_scrub_reservations 2019-12-21T15:55:06.828 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=0 2019-12-21T15:55:06.828 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 0 -gt 4 ']' 2019-12-21T15:55:06.828 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 14 + 0 2019-12-21T15:55:06.829 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=14 2019-12-21T15:55:06.830 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:133: TEST_recover_unexpected: '[' 14 = 0 ']' 2019-12-21T15:55:06.830 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:136: TEST_recover_unexpected: '[' 0 -gt 3 ']' 2019-12-21T15:55:06.830 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: expr 494 + 14 2019-12-21T15:55:06.831 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: total=508 2019-12-21T15:55:06.831 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: expr 2 '*' 2 2019-12-21T15:55:06.832 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: sleep 4 2019-12-21T15:55:10.834 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:120: TEST_recover_unexpected: true 2019-12-21T15:55:10.834 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:122: TEST_recover_unexpected: pass=0 2019-12-21T15:55:10.835 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: expr 6 - 1 2019-12-21T15:55:10.836 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: seq 0 5 2019-12-21T15:55:10.837 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:55:10.838 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.0 2019-12-21T15:55:10.839 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2019-12-21T15:55:10.839 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2019-12-21T15:55:10.839 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:55:10.840 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:55:10.840 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:55:10.840 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.0.asok 2019-12-21T15:55:10.840 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:55:10.840 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.0.asok dump_scrub_reservations 2019-12-21T15:55:10.939 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:55:10.940 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 0, 2019-12-21T15:55:10.940 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 4, 2019-12-21T15:55:10.940 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:55:10.940 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:55:10.949 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:55:10.950 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.0 2019-12-21T15:55:10.950 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2019-12-21T15:55:10.950 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2019-12-21T15:55:10.950 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:55:10.950 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:55:10.950 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:55:10.950 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.0.asok 2019-12-21T15:55:10.951 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:55:10.951 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.0.asok dump_scrub_reservations 2019-12-21T15:55:11.048 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=4 2019-12-21T15:55:11.048 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2019-12-21T15:55:11.048 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 0 + 4 2019-12-21T15:55:11.049 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=4 2019-12-21T15:55:11.049 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:55:11.050 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.1 2019-12-21T15:55:11.050 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2019-12-21T15:55:11.051 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2019-12-21T15:55:11.051 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:55:11.051 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:55:11.051 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:55:11.051 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.1.asok 2019-12-21T15:55:11.052 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:55:11.052 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.1.asok dump_scrub_reservations 2019-12-21T15:55:11.140 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:55:11.140 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 1, 2019-12-21T15:55:11.140 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 2, 2019-12-21T15:55:11.140 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:55:11.140 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:55:11.148 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:55:11.149 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.1 2019-12-21T15:55:11.149 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2019-12-21T15:55:11.149 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2019-12-21T15:55:11.149 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:55:11.149 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:55:11.150 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:55:11.150 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.1.asok 2019-12-21T15:55:11.150 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:55:11.150 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.1.asok dump_scrub_reservations 2019-12-21T15:55:11.244 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=3 2019-12-21T15:55:11.245 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2019-12-21T15:55:11.245 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 4 + 3 2019-12-21T15:55:11.246 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=7 2019-12-21T15:55:11.246 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:55:11.247 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.2 2019-12-21T15:55:11.247 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2019-12-21T15:55:11.248 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2019-12-21T15:55:11.248 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:55:11.248 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:55:11.248 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:55:11.248 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.2.asok 2019-12-21T15:55:11.248 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:55:11.249 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.2.asok dump_scrub_reservations 2019-12-21T15:55:11.335 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:55:11.336 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 0, 2019-12-21T15:55:11.336 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 2, 2019-12-21T15:55:11.336 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:55:11.336 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:55:11.345 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:55:11.345 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.2 2019-12-21T15:55:11.345 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2019-12-21T15:55:11.345 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2019-12-21T15:55:11.346 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:55:11.346 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:55:11.346 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:55:11.346 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.2.asok 2019-12-21T15:55:11.346 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:55:11.346 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.2.asok dump_scrub_reservations 2019-12-21T15:55:11.440 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=2 2019-12-21T15:55:11.440 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 2 -gt 4 ']' 2019-12-21T15:55:11.441 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 7 + 2 2019-12-21T15:55:11.441 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=9 2019-12-21T15:55:11.442 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:55:11.442 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.3 2019-12-21T15:55:11.442 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2019-12-21T15:55:11.443 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2019-12-21T15:55:11.443 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:55:11.444 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:55:11.444 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:55:11.444 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.3.asok 2019-12-21T15:55:11.444 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:55:11.444 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.3.asok dump_scrub_reservations 2019-12-21T15:55:11.531 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:55:11.531 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 1, 2019-12-21T15:55:11.531 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 1, 2019-12-21T15:55:11.531 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:55:11.532 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:55:11.540 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:55:11.540 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.3 2019-12-21T15:55:11.540 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2019-12-21T15:55:11.541 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2019-12-21T15:55:11.541 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:55:11.541 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:55:11.541 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:55:11.542 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.3.asok 2019-12-21T15:55:11.542 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:55:11.542 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.3.asok dump_scrub_reservations 2019-12-21T15:55:11.636 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=2 2019-12-21T15:55:11.636 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 2 -gt 4 ']' 2019-12-21T15:55:11.637 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 9 + 2 2019-12-21T15:55:11.637 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=11 2019-12-21T15:55:11.637 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:55:11.639 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.4 2019-12-21T15:55:11.639 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2019-12-21T15:55:11.639 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2019-12-21T15:55:11.639 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:55:11.639 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:55:11.640 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:55:11.640 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.4.asok 2019-12-21T15:55:11.640 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:55:11.640 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.4.asok dump_scrub_reservations 2019-12-21T15:55:11.727 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:55:11.727 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 3, 2019-12-21T15:55:11.727 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 1, 2019-12-21T15:55:11.728 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:55:11.728 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:55:11.736 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:55:11.736 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.4 2019-12-21T15:55:11.736 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2019-12-21T15:55:11.736 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2019-12-21T15:55:11.737 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:55:11.737 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:55:11.737 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:55:11.737 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.4.asok 2019-12-21T15:55:11.737 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:55:11.737 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.4.asok dump_scrub_reservations 2019-12-21T15:55:11.832 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=4 2019-12-21T15:55:11.832 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2019-12-21T15:55:11.833 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 11 + 4 2019-12-21T15:55:11.833 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=15 2019-12-21T15:55:11.833 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:55:11.835 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.5 2019-12-21T15:55:11.835 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2019-12-21T15:55:11.835 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2019-12-21T15:55:11.835 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:55:11.835 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:55:11.835 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:55:11.836 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.5.asok 2019-12-21T15:55:11.836 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:55:11.836 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.5.asok dump_scrub_reservations 2019-12-21T15:55:11.923 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:55:11.924 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 0, 2019-12-21T15:55:11.924 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 0, 2019-12-21T15:55:11.924 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:55:11.924 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:55:11.932 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:55:11.932 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.5 2019-12-21T15:55:11.933 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2019-12-21T15:55:11.933 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2019-12-21T15:55:11.933 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:55:11.933 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:55:11.933 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:55:11.933 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.5.asok 2019-12-21T15:55:11.934 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:55:11.934 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.5.asok dump_scrub_reservations 2019-12-21T15:55:12.027 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=0 2019-12-21T15:55:12.028 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 0 -gt 4 ']' 2019-12-21T15:55:12.028 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 15 + 0 2019-12-21T15:55:12.029 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=15 2019-12-21T15:55:12.029 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:133: TEST_recover_unexpected: '[' 15 = 0 ']' 2019-12-21T15:55:12.029 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:136: TEST_recover_unexpected: '[' 0 -gt 3 ']' 2019-12-21T15:55:12.030 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: expr 508 + 15 2019-12-21T15:55:12.030 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: total=523 2019-12-21T15:55:12.031 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: expr 2 '*' 2 2019-12-21T15:55:12.032 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: sleep 4 2019-12-21T15:55:16.033 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:120: TEST_recover_unexpected: true 2019-12-21T15:55:16.034 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:122: TEST_recover_unexpected: pass=0 2019-12-21T15:55:16.035 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: expr 6 - 1 2019-12-21T15:55:16.036 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: seq 0 5 2019-12-21T15:55:16.037 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:55:16.038 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.0 2019-12-21T15:55:16.039 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2019-12-21T15:55:16.039 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2019-12-21T15:55:16.039 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:55:16.039 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:55:16.039 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:55:16.039 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.0.asok 2019-12-21T15:55:16.040 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:55:16.040 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.0.asok dump_scrub_reservations 2019-12-21T15:55:16.137 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:55:16.137 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 0, 2019-12-21T15:55:16.137 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 4, 2019-12-21T15:55:16.138 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:55:16.138 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:55:16.147 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:55:16.147 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.0 2019-12-21T15:55:16.147 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2019-12-21T15:55:16.147 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2019-12-21T15:55:16.148 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:55:16.148 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:55:16.148 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:55:16.148 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.0.asok 2019-12-21T15:55:16.148 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:55:16.148 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.0.asok dump_scrub_reservations 2019-12-21T15:55:16.242 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=4 2019-12-21T15:55:16.242 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2019-12-21T15:55:16.243 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 0 + 4 2019-12-21T15:55:16.243 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=4 2019-12-21T15:55:16.243 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:55:16.245 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.1 2019-12-21T15:55:16.245 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2019-12-21T15:55:16.245 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2019-12-21T15:55:16.246 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:55:16.246 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:55:16.246 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:55:16.246 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.1.asok 2019-12-21T15:55:16.246 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:55:16.246 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.1.asok dump_scrub_reservations 2019-12-21T15:55:16.334 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:55:16.334 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 2, 2019-12-21T15:55:16.334 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 1, 2019-12-21T15:55:16.335 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:55:16.335 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:55:16.343 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:55:16.343 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.1 2019-12-21T15:55:16.343 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2019-12-21T15:55:16.343 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2019-12-21T15:55:16.343 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:55:16.344 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:55:16.344 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:55:16.344 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.1.asok 2019-12-21T15:55:16.344 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:55:16.344 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.1.asok dump_scrub_reservations 2019-12-21T15:55:16.438 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=3 2019-12-21T15:55:16.439 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2019-12-21T15:55:16.439 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 4 + 3 2019-12-21T15:55:16.440 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=7 2019-12-21T15:55:16.440 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:55:16.441 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.2 2019-12-21T15:55:16.441 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2019-12-21T15:55:16.442 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2019-12-21T15:55:16.442 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:55:16.442 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:55:16.442 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:55:16.442 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.2.asok 2019-12-21T15:55:16.443 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:55:16.443 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.2.asok dump_scrub_reservations 2019-12-21T15:55:16.529 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:55:16.530 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 0, 2019-12-21T15:55:16.530 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 4, 2019-12-21T15:55:16.530 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:55:16.530 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:55:16.538 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:55:16.538 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.2 2019-12-21T15:55:16.538 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2019-12-21T15:55:16.539 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2019-12-21T15:55:16.539 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:55:16.539 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:55:16.539 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:55:16.539 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.2.asok 2019-12-21T15:55:16.539 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:55:16.539 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.2.asok dump_scrub_reservations 2019-12-21T15:55:16.634 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=4 2019-12-21T15:55:16.634 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2019-12-21T15:55:16.635 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 7 + 4 2019-12-21T15:55:16.635 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=11 2019-12-21T15:55:16.635 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:55:16.637 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.3 2019-12-21T15:55:16.637 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2019-12-21T15:55:16.637 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2019-12-21T15:55:16.637 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:55:16.638 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:55:16.638 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:55:16.638 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.3.asok 2019-12-21T15:55:16.638 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:55:16.638 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.3.asok dump_scrub_reservations 2019-12-21T15:55:16.725 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:55:16.726 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 1, 2019-12-21T15:55:16.726 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 1, 2019-12-21T15:55:16.726 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:55:16.726 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:55:16.734 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:55:16.734 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.3 2019-12-21T15:55:16.735 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2019-12-21T15:55:16.735 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2019-12-21T15:55:16.735 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:55:16.735 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:55:16.735 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:55:16.735 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.3.asok 2019-12-21T15:55:16.735 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:55:16.736 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.3.asok dump_scrub_reservations 2019-12-21T15:55:16.830 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=2 2019-12-21T15:55:16.830 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 2 -gt 4 ']' 2019-12-21T15:55:16.831 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 11 + 2 2019-12-21T15:55:16.831 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=13 2019-12-21T15:55:16.831 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:55:16.833 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.4 2019-12-21T15:55:16.833 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2019-12-21T15:55:16.833 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2019-12-21T15:55:16.833 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:55:16.833 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:55:16.833 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:55:16.834 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.4.asok 2019-12-21T15:55:16.834 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:55:16.834 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.4.asok dump_scrub_reservations 2019-12-21T15:55:16.922 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:55:16.922 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 3, 2019-12-21T15:55:16.922 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 1, 2019-12-21T15:55:16.922 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:55:16.922 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:55:16.930 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:55:16.930 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.4 2019-12-21T15:55:16.931 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2019-12-21T15:55:16.931 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2019-12-21T15:55:16.931 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:55:16.931 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:55:16.931 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:55:16.931 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.4.asok 2019-12-21T15:55:16.932 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:55:16.932 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.4.asok dump_scrub_reservations 2019-12-21T15:55:17.026 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=4 2019-12-21T15:55:17.026 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2019-12-21T15:55:17.026 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 13 + 4 2019-12-21T15:55:17.027 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=17 2019-12-21T15:55:17.027 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:55:17.028 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.5 2019-12-21T15:55:17.029 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2019-12-21T15:55:17.029 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2019-12-21T15:55:17.029 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:55:17.029 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:55:17.029 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:55:17.029 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.5.asok 2019-12-21T15:55:17.030 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:55:17.030 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.5.asok dump_scrub_reservations 2019-12-21T15:55:17.118 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:55:17.118 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 0, 2019-12-21T15:55:17.118 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 1, 2019-12-21T15:55:17.119 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:55:17.119 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:55:17.127 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:55:17.128 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.5 2019-12-21T15:55:17.128 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2019-12-21T15:55:17.128 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2019-12-21T15:55:17.128 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:55:17.128 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:55:17.128 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:55:17.128 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.5.asok 2019-12-21T15:55:17.129 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:55:17.129 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.5.asok dump_scrub_reservations 2019-12-21T15:55:17.223 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=1 2019-12-21T15:55:17.223 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 1 -gt 4 ']' 2019-12-21T15:55:17.224 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 17 + 1 2019-12-21T15:55:17.225 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=18 2019-12-21T15:55:17.225 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:133: TEST_recover_unexpected: '[' 18 = 0 ']' 2019-12-21T15:55:17.225 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:136: TEST_recover_unexpected: '[' 0 -gt 3 ']' 2019-12-21T15:55:17.225 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: expr 523 + 18 2019-12-21T15:55:17.226 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: total=541 2019-12-21T15:55:17.227 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: expr 2 '*' 2 2019-12-21T15:55:17.227 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: sleep 4 2019-12-21T15:55:21.229 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:120: TEST_recover_unexpected: true 2019-12-21T15:55:21.229 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:122: TEST_recover_unexpected: pass=0 2019-12-21T15:55:21.230 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: expr 6 - 1 2019-12-21T15:55:21.232 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: seq 0 5 2019-12-21T15:55:21.232 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:55:21.234 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.0 2019-12-21T15:55:21.234 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2019-12-21T15:55:21.234 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2019-12-21T15:55:21.235 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:55:21.235 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:55:21.235 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:55:21.236 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.0.asok 2019-12-21T15:55:21.236 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:55:21.236 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.0.asok dump_scrub_reservations 2019-12-21T15:55:21.332 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:55:21.333 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 0, 2019-12-21T15:55:21.333 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 4, 2019-12-21T15:55:21.333 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:55:21.333 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:55:21.342 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:55:21.342 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.0 2019-12-21T15:55:21.342 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2019-12-21T15:55:21.342 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2019-12-21T15:55:21.343 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:55:21.343 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:55:21.343 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:55:21.343 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.0.asok 2019-12-21T15:55:21.343 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:55:21.343 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.0.asok dump_scrub_reservations 2019-12-21T15:55:21.439 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=4 2019-12-21T15:55:21.439 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2019-12-21T15:55:21.439 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 0 + 4 2019-12-21T15:55:21.440 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=4 2019-12-21T15:55:21.440 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:55:21.441 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.1 2019-12-21T15:55:21.442 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2019-12-21T15:55:21.442 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2019-12-21T15:55:21.442 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:55:21.442 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:55:21.442 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:55:21.442 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.1.asok 2019-12-21T15:55:21.443 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:55:21.443 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.1.asok dump_scrub_reservations 2019-12-21T15:55:21.529 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:55:21.529 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 2, 2019-12-21T15:55:21.529 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 0, 2019-12-21T15:55:21.529 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:55:21.529 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:55:21.538 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:55:21.538 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.1 2019-12-21T15:55:21.538 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2019-12-21T15:55:21.538 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2019-12-21T15:55:21.539 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:55:21.540 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:55:21.540 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:55:21.540 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.1.asok 2019-12-21T15:55:21.540 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:55:21.540 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.1.asok dump_scrub_reservations 2019-12-21T15:55:21.633 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=2 2019-12-21T15:55:21.633 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 2 -gt 4 ']' 2019-12-21T15:55:21.634 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 4 + 2 2019-12-21T15:55:21.635 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=6 2019-12-21T15:55:21.635 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:55:21.636 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.2 2019-12-21T15:55:21.636 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2019-12-21T15:55:21.636 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2019-12-21T15:55:21.637 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:55:21.637 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:55:21.637 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:55:21.637 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.2.asok 2019-12-21T15:55:21.637 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:55:21.638 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.2.asok dump_scrub_reservations 2019-12-21T15:55:21.729 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:55:21.729 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 0, 2019-12-21T15:55:21.729 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 3, 2019-12-21T15:55:21.729 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:55:21.729 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:55:21.734 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:55:21.734 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.2 2019-12-21T15:55:21.734 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2019-12-21T15:55:21.734 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2019-12-21T15:55:21.734 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:55:21.734 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:55:21.735 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:55:21.735 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.2.asok 2019-12-21T15:55:21.735 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:55:21.735 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.2.asok dump_scrub_reservations 2019-12-21T15:55:21.829 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=3 2019-12-21T15:55:21.829 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2019-12-21T15:55:21.830 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 6 + 3 2019-12-21T15:55:21.830 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=9 2019-12-21T15:55:21.831 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:55:21.832 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.3 2019-12-21T15:55:21.832 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2019-12-21T15:55:21.832 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2019-12-21T15:55:21.832 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:55:21.832 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:55:21.833 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:55:21.833 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.3.asok 2019-12-21T15:55:21.833 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:55:21.833 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.3.asok dump_scrub_reservations 2019-12-21T15:55:21.922 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:55:21.922 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 1, 2019-12-21T15:55:21.922 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 1, 2019-12-21T15:55:21.922 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:55:21.922 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:55:21.931 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:55:21.931 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.3 2019-12-21T15:55:21.931 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2019-12-21T15:55:21.931 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2019-12-21T15:55:21.931 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:55:21.931 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:55:21.932 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:55:21.932 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.3.asok 2019-12-21T15:55:21.932 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:55:21.932 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.3.asok dump_scrub_reservations 2019-12-21T15:55:22.026 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=2 2019-12-21T15:55:22.027 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 2 -gt 4 ']' 2019-12-21T15:55:22.027 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 9 + 2 2019-12-21T15:55:22.028 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=11 2019-12-21T15:55:22.028 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:55:22.029 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.4 2019-12-21T15:55:22.029 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2019-12-21T15:55:22.029 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2019-12-21T15:55:22.030 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:55:22.030 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:55:22.030 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:55:22.030 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.4.asok 2019-12-21T15:55:22.031 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:55:22.031 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.4.asok dump_scrub_reservations 2019-12-21T15:55:22.120 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:55:22.120 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 2, 2019-12-21T15:55:22.120 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 1, 2019-12-21T15:55:22.120 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:55:22.121 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:55:22.129 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:55:22.129 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.4 2019-12-21T15:55:22.129 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2019-12-21T15:55:22.129 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2019-12-21T15:55:22.130 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:55:22.130 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:55:22.130 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:55:22.130 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.4.asok 2019-12-21T15:55:22.130 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:55:22.130 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.4.asok dump_scrub_reservations 2019-12-21T15:55:22.225 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=3 2019-12-21T15:55:22.225 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2019-12-21T15:55:22.226 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 11 + 3 2019-12-21T15:55:22.228 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=14 2019-12-21T15:55:22.228 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:55:22.228 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.5 2019-12-21T15:55:22.228 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2019-12-21T15:55:22.229 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2019-12-21T15:55:22.229 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:55:22.229 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:55:22.229 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:55:22.229 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.5.asok 2019-12-21T15:55:22.230 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:55:22.230 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.5.asok dump_scrub_reservations 2019-12-21T15:55:22.317 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:55:22.317 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 0, 2019-12-21T15:55:22.317 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 1, 2019-12-21T15:55:22.317 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:55:22.317 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:55:22.325 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:55:22.325 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.5 2019-12-21T15:55:22.325 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2019-12-21T15:55:22.325 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2019-12-21T15:55:22.326 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:55:22.327 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:55:22.327 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:55:22.327 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.5.asok 2019-12-21T15:55:22.327 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:55:22.327 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.5.asok dump_scrub_reservations 2019-12-21T15:55:22.421 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=1 2019-12-21T15:55:22.421 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 1 -gt 4 ']' 2019-12-21T15:55:22.422 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 14 + 1 2019-12-21T15:55:22.423 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=15 2019-12-21T15:55:22.423 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:133: TEST_recover_unexpected: '[' 15 = 0 ']' 2019-12-21T15:55:22.423 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:136: TEST_recover_unexpected: '[' 0 -gt 3 ']' 2019-12-21T15:55:22.423 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: expr 541 + 15 2019-12-21T15:55:22.424 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: total=556 2019-12-21T15:55:22.425 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: expr 2 '*' 2 2019-12-21T15:55:22.425 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: sleep 4 2019-12-21T15:55:26.427 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:120: TEST_recover_unexpected: true 2019-12-21T15:55:26.427 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:122: TEST_recover_unexpected: pass=0 2019-12-21T15:55:26.428 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: expr 6 - 1 2019-12-21T15:55:26.430 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: seq 0 5 2019-12-21T15:55:26.431 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:55:26.432 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.0 2019-12-21T15:55:26.432 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2019-12-21T15:55:26.433 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2019-12-21T15:55:26.433 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:55:26.433 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:55:26.433 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:55:26.433 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.0.asok 2019-12-21T15:55:26.434 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:55:26.434 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.0.asok dump_scrub_reservations 2019-12-21T15:55:26.530 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:55:26.711 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 0, 2019-12-21T15:55:26.711 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 4, 2019-12-21T15:55:26.711 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:55:26.711 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:55:26.714 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:55:26.714 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.0 2019-12-21T15:55:26.714 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2019-12-21T15:55:26.715 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2019-12-21T15:55:26.715 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:55:26.715 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:55:26.715 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:55:26.715 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.0.asok 2019-12-21T15:55:26.715 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:55:26.715 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.0.asok dump_scrub_reservations 2019-12-21T15:55:26.716 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=4 2019-12-21T15:55:26.716 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2019-12-21T15:55:26.716 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 0 + 4 2019-12-21T15:55:26.716 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=4 2019-12-21T15:55:26.716 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:55:26.716 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.1 2019-12-21T15:55:26.716 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2019-12-21T15:55:26.717 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2019-12-21T15:55:26.717 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:55:26.717 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:55:26.717 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:55:26.717 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.1.asok 2019-12-21T15:55:26.717 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:55:26.717 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.1.asok dump_scrub_reservations 2019-12-21T15:55:26.727 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:55:26.727 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 2, 2019-12-21T15:55:26.727 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 0, 2019-12-21T15:55:26.728 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:55:26.728 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:55:26.736 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:55:26.736 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.1 2019-12-21T15:55:26.736 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2019-12-21T15:55:26.736 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2019-12-21T15:55:26.737 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:55:26.737 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:55:26.737 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:55:26.737 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.1.asok 2019-12-21T15:55:26.737 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:55:26.737 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.1.asok dump_scrub_reservations 2019-12-21T15:55:26.831 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=2 2019-12-21T15:55:26.832 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 2 -gt 4 ']' 2019-12-21T15:55:26.832 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 4 + 2 2019-12-21T15:55:26.833 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=6 2019-12-21T15:55:26.833 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:55:26.835 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.2 2019-12-21T15:55:26.835 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2019-12-21T15:55:26.835 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2019-12-21T15:55:26.835 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:55:26.835 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:55:26.836 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:55:26.836 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.2.asok 2019-12-21T15:55:26.836 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:55:26.836 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.2.asok dump_scrub_reservations 2019-12-21T15:55:26.923 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:55:26.923 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 0, 2019-12-21T15:55:26.923 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 4, 2019-12-21T15:55:26.924 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:55:26.925 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:55:26.934 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:55:26.934 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.2 2019-12-21T15:55:26.935 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2019-12-21T15:55:26.935 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2019-12-21T15:55:26.935 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:55:26.935 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:55:26.936 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:55:26.936 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.2.asok 2019-12-21T15:55:26.936 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:55:26.936 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.2.asok dump_scrub_reservations 2019-12-21T15:55:27.028 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=4 2019-12-21T15:55:27.028 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2019-12-21T15:55:27.028 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 6 + 4 2019-12-21T15:55:27.029 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=10 2019-12-21T15:55:27.029 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:55:27.032 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.3 2019-12-21T15:55:27.033 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2019-12-21T15:55:27.033 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2019-12-21T15:55:27.033 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:55:27.033 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:55:27.034 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:55:27.034 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.3.asok 2019-12-21T15:55:27.034 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:55:27.034 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.3.asok dump_scrub_reservations 2019-12-21T15:55:27.120 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:55:27.121 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 1, 2019-12-21T15:55:27.121 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 1, 2019-12-21T15:55:27.121 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:55:27.121 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:55:27.129 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:55:27.130 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.3 2019-12-21T15:55:27.130 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2019-12-21T15:55:27.130 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2019-12-21T15:55:27.130 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:55:27.130 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:55:27.130 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:55:27.131 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.3.asok 2019-12-21T15:55:27.131 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:55:27.131 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.3.asok dump_scrub_reservations 2019-12-21T15:55:27.224 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=2 2019-12-21T15:55:27.224 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 2 -gt 4 ']' 2019-12-21T15:55:27.225 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 10 + 2 2019-12-21T15:55:27.226 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=12 2019-12-21T15:55:27.226 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:55:27.227 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.4 2019-12-21T15:55:27.228 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2019-12-21T15:55:27.228 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2019-12-21T15:55:27.228 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:55:27.228 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:55:27.228 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:55:27.228 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.4.asok 2019-12-21T15:55:27.229 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:55:27.229 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.4.asok dump_scrub_reservations 2019-12-21T15:55:27.315 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:55:27.315 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 3, 2019-12-21T15:55:27.315 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 1, 2019-12-21T15:55:27.316 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:55:27.316 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:55:27.324 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:55:27.324 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.4 2019-12-21T15:55:27.324 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2019-12-21T15:55:27.325 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2019-12-21T15:55:27.325 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:55:27.325 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:55:27.325 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:55:27.325 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.4.asok 2019-12-21T15:55:27.325 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:55:27.326 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.4.asok dump_scrub_reservations 2019-12-21T15:55:27.420 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=4 2019-12-21T15:55:27.420 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2019-12-21T15:55:27.420 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 12 + 4 2019-12-21T15:55:27.421 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=16 2019-12-21T15:55:27.421 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:55:27.423 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.5 2019-12-21T15:55:27.423 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2019-12-21T15:55:27.423 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2019-12-21T15:55:27.423 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:55:27.424 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:55:27.424 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:55:27.424 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.5.asok 2019-12-21T15:55:27.424 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:55:27.424 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.5.asok dump_scrub_reservations 2019-12-21T15:55:27.511 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:55:27.512 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 0, 2019-12-21T15:55:27.512 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 2, 2019-12-21T15:55:27.512 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:55:27.512 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:55:27.521 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:55:27.521 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.5 2019-12-21T15:55:27.521 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2019-12-21T15:55:27.522 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2019-12-21T15:55:27.522 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:55:27.522 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:55:27.522 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:55:27.522 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.5.asok 2019-12-21T15:55:27.523 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:55:27.523 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.5.asok dump_scrub_reservations 2019-12-21T15:55:27.617 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=2 2019-12-21T15:55:27.617 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 2 -gt 4 ']' 2019-12-21T15:55:27.618 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 16 + 2 2019-12-21T15:55:27.619 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=18 2019-12-21T15:55:27.619 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:133: TEST_recover_unexpected: '[' 18 = 0 ']' 2019-12-21T15:55:27.619 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:136: TEST_recover_unexpected: '[' 0 -gt 3 ']' 2019-12-21T15:55:27.620 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: expr 556 + 18 2019-12-21T15:55:27.620 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: total=574 2019-12-21T15:55:27.621 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: expr 2 '*' 2 2019-12-21T15:55:27.621 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: sleep 4 2019-12-21T15:55:31.623 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:120: TEST_recover_unexpected: true 2019-12-21T15:55:31.623 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:122: TEST_recover_unexpected: pass=0 2019-12-21T15:55:31.624 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: expr 6 - 1 2019-12-21T15:55:31.625 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: seq 0 5 2019-12-21T15:55:31.626 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:55:31.627 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.0 2019-12-21T15:55:31.627 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2019-12-21T15:55:31.628 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2019-12-21T15:55:31.628 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:55:31.629 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:55:31.629 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:55:31.629 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.0.asok 2019-12-21T15:55:31.629 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:55:31.629 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.0.asok dump_scrub_reservations 2019-12-21T15:55:31.726 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:55:31.726 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 0, 2019-12-21T15:55:31.726 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 4, 2019-12-21T15:55:31.726 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:55:31.726 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:55:31.735 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:55:31.735 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.0 2019-12-21T15:55:31.735 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2019-12-21T15:55:31.735 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2019-12-21T15:55:31.736 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:55:31.736 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:55:31.736 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:55:31.736 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.0.asok 2019-12-21T15:55:31.736 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:55:31.736 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.0.asok dump_scrub_reservations 2019-12-21T15:55:31.832 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=4 2019-12-21T15:55:31.832 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2019-12-21T15:55:31.833 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 0 + 4 2019-12-21T15:55:31.833 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=4 2019-12-21T15:55:31.834 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:55:31.835 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.1 2019-12-21T15:55:31.835 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2019-12-21T15:55:31.835 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2019-12-21T15:55:31.835 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:55:31.835 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:55:31.835 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:55:31.835 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.1.asok 2019-12-21T15:55:31.836 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:55:31.836 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.1.asok dump_scrub_reservations 2019-12-21T15:55:31.923 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:55:31.924 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 3, 2019-12-21T15:55:31.924 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 0, 2019-12-21T15:55:31.924 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:55:31.924 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:55:31.932 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:55:31.932 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.1 2019-12-21T15:55:31.932 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2019-12-21T15:55:31.933 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2019-12-21T15:55:31.933 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:55:31.933 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:55:31.933 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:55:31.933 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.1.asok 2019-12-21T15:55:31.933 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:55:31.934 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.1.asok dump_scrub_reservations 2019-12-21T15:55:32.027 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=3 2019-12-21T15:55:32.027 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2019-12-21T15:55:32.028 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 4 + 3 2019-12-21T15:55:32.029 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=7 2019-12-21T15:55:32.029 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:55:32.030 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.2 2019-12-21T15:55:32.030 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2019-12-21T15:55:32.030 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2019-12-21T15:55:32.030 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:55:32.031 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:55:32.031 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:55:32.031 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.2.asok 2019-12-21T15:55:32.031 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:55:32.032 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.2.asok dump_scrub_reservations 2019-12-21T15:55:32.119 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:55:32.119 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 0, 2019-12-21T15:55:32.119 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 4, 2019-12-21T15:55:32.119 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:55:32.119 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:55:32.131 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:55:32.131 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.2 2019-12-21T15:55:32.131 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2019-12-21T15:55:32.131 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2019-12-21T15:55:32.132 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:55:32.132 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:55:32.132 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:55:32.132 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.2.asok 2019-12-21T15:55:32.132 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:55:32.132 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.2.asok dump_scrub_reservations 2019-12-21T15:55:32.224 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=4 2019-12-21T15:55:32.224 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2019-12-21T15:55:32.224 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 7 + 4 2019-12-21T15:55:32.225 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=11 2019-12-21T15:55:32.225 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:55:32.227 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.3 2019-12-21T15:55:32.227 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2019-12-21T15:55:32.227 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2019-12-21T15:55:32.227 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:55:32.227 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:55:32.227 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:55:32.227 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.3.asok 2019-12-21T15:55:32.228 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:55:32.228 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.3.asok dump_scrub_reservations 2019-12-21T15:55:32.317 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:55:32.317 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 0, 2019-12-21T15:55:32.317 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 0, 2019-12-21T15:55:32.317 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:55:32.317 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:55:32.326 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:55:32.326 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.3 2019-12-21T15:55:32.326 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2019-12-21T15:55:32.326 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2019-12-21T15:55:32.326 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:55:32.327 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:55:32.327 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:55:32.327 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.3.asok 2019-12-21T15:55:32.327 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:55:32.327 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.3.asok dump_scrub_reservations 2019-12-21T15:55:32.421 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=0 2019-12-21T15:55:32.421 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 0 -gt 4 ']' 2019-12-21T15:55:32.422 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 11 + 0 2019-12-21T15:55:32.425 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=11 2019-12-21T15:55:32.425 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:55:32.425 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.4 2019-12-21T15:55:32.425 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2019-12-21T15:55:32.426 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2019-12-21T15:55:32.426 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:55:32.426 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:55:32.426 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:55:32.426 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.4.asok 2019-12-21T15:55:32.426 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:55:32.426 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.4.asok dump_scrub_reservations 2019-12-21T15:55:32.512 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:55:32.513 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 2, 2019-12-21T15:55:32.513 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 0, 2019-12-21T15:55:32.513 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:55:32.513 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:55:32.521 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:55:32.521 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.4 2019-12-21T15:55:32.522 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2019-12-21T15:55:32.522 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2019-12-21T15:55:32.522 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:55:32.522 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:55:32.522 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:55:32.522 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.4.asok 2019-12-21T15:55:32.523 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:55:32.523 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.4.asok dump_scrub_reservations 2019-12-21T15:55:32.616 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=2 2019-12-21T15:55:32.617 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 2 -gt 4 ']' 2019-12-21T15:55:32.617 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 11 + 2 2019-12-21T15:55:32.618 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=13 2019-12-21T15:55:32.618 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:55:32.619 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.5 2019-12-21T15:55:32.619 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2019-12-21T15:55:32.620 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2019-12-21T15:55:32.620 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:55:32.620 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:55:32.620 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:55:32.620 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.5.asok 2019-12-21T15:55:32.620 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:55:32.620 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.5.asok dump_scrub_reservations 2019-12-21T15:55:32.708 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:55:32.708 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 0, 2019-12-21T15:55:32.709 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 3, 2019-12-21T15:55:32.709 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:55:32.709 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:55:32.717 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:55:32.717 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.5 2019-12-21T15:55:32.718 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2019-12-21T15:55:32.718 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2019-12-21T15:55:32.718 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:55:32.718 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:55:32.721 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:55:32.721 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.5.asok 2019-12-21T15:55:32.721 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:55:32.721 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.5.asok dump_scrub_reservations 2019-12-21T15:55:32.813 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=3 2019-12-21T15:55:32.813 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2019-12-21T15:55:32.814 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 13 + 3 2019-12-21T15:55:32.814 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=16 2019-12-21T15:55:32.814 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:133: TEST_recover_unexpected: '[' 16 = 0 ']' 2019-12-21T15:55:32.815 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:136: TEST_recover_unexpected: '[' 0 -gt 3 ']' 2019-12-21T15:55:32.815 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: expr 574 + 16 2019-12-21T15:55:32.816 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: total=590 2019-12-21T15:55:32.816 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: expr 2 '*' 2 2019-12-21T15:55:32.817 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: sleep 4 2019-12-21T15:55:36.818 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:120: TEST_recover_unexpected: true 2019-12-21T15:55:36.819 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:122: TEST_recover_unexpected: pass=0 2019-12-21T15:55:36.820 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: expr 6 - 1 2019-12-21T15:55:36.821 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: seq 0 5 2019-12-21T15:55:36.822 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:55:36.824 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.0 2019-12-21T15:55:36.824 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2019-12-21T15:55:36.824 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2019-12-21T15:55:36.824 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:55:36.824 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:55:36.825 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:55:36.825 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.0.asok 2019-12-21T15:55:36.825 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:55:36.825 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.0.asok dump_scrub_reservations 2019-12-21T15:55:36.922 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:55:36.922 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 0, 2019-12-21T15:55:36.923 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 1, 2019-12-21T15:55:36.923 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:55:36.923 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:55:36.931 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:55:36.931 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.0 2019-12-21T15:55:36.932 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2019-12-21T15:55:36.932 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2019-12-21T15:55:36.932 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:55:36.932 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:55:36.932 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:55:36.932 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.0.asok 2019-12-21T15:55:36.933 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:55:36.933 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.0.asok dump_scrub_reservations 2019-12-21T15:55:37.028 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=1 2019-12-21T15:55:37.028 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 1 -gt 4 ']' 2019-12-21T15:55:37.029 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 0 + 1 2019-12-21T15:55:37.030 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=1 2019-12-21T15:55:37.030 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:55:37.031 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.1 2019-12-21T15:55:37.031 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2019-12-21T15:55:37.031 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2019-12-21T15:55:37.031 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:55:37.032 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:55:37.032 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:55:37.032 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.1.asok 2019-12-21T15:55:37.032 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:55:37.033 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.1.asok dump_scrub_reservations 2019-12-21T15:55:37.121 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:55:37.121 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 1, 2019-12-21T15:55:37.121 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 0, 2019-12-21T15:55:37.121 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:55:37.121 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:55:37.130 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:55:37.130 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.1 2019-12-21T15:55:37.130 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2019-12-21T15:55:37.130 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2019-12-21T15:55:37.130 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:55:37.131 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:55:37.131 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:55:37.131 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.1.asok 2019-12-21T15:55:37.131 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:55:37.131 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.1.asok dump_scrub_reservations 2019-12-21T15:55:37.225 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=1 2019-12-21T15:55:37.225 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 1 -gt 4 ']' 2019-12-21T15:55:37.225 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 1 + 1 2019-12-21T15:55:37.226 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=2 2019-12-21T15:55:37.226 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:55:37.227 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.2 2019-12-21T15:55:37.228 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2019-12-21T15:55:37.228 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2019-12-21T15:55:37.228 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:55:37.228 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:55:37.228 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:55:37.228 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.2.asok 2019-12-21T15:55:37.231 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:55:37.231 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.2.asok dump_scrub_reservations 2019-12-21T15:55:37.316 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:55:37.316 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 0, 2019-12-21T15:55:37.316 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 2, 2019-12-21T15:55:37.317 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:55:37.317 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:55:37.325 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:55:37.325 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.2 2019-12-21T15:55:37.325 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2019-12-21T15:55:37.325 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2019-12-21T15:55:37.325 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:55:37.326 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:55:37.326 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:55:37.326 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.2.asok 2019-12-21T15:55:37.326 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:55:37.326 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.2.asok dump_scrub_reservations 2019-12-21T15:55:37.419 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=2 2019-12-21T15:55:37.420 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 2 -gt 4 ']' 2019-12-21T15:55:37.420 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 2 + 2 2019-12-21T15:55:37.421 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=4 2019-12-21T15:55:37.421 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:55:37.423 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.3 2019-12-21T15:55:37.423 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2019-12-21T15:55:37.423 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2019-12-21T15:55:37.423 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:55:37.423 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:55:37.423 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:55:37.424 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.3.asok 2019-12-21T15:55:37.424 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:55:37.424 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.3.asok dump_scrub_reservations 2019-12-21T15:55:37.510 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:55:37.510 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 0, 2019-12-21T15:55:37.511 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 0, 2019-12-21T15:55:37.511 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:55:37.511 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:55:37.519 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:55:37.519 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.3 2019-12-21T15:55:37.519 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2019-12-21T15:55:37.519 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2019-12-21T15:55:37.520 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:55:37.520 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:55:37.520 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:55:37.520 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.3.asok 2019-12-21T15:55:37.520 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:55:37.520 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.3.asok dump_scrub_reservations 2019-12-21T15:55:37.614 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=0 2019-12-21T15:55:37.615 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 0 -gt 4 ']' 2019-12-21T15:55:37.615 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 4 + 0 2019-12-21T15:55:37.616 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=4 2019-12-21T15:55:37.616 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:55:37.617 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.4 2019-12-21T15:55:37.618 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2019-12-21T15:55:37.618 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2019-12-21T15:55:37.618 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:55:37.618 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:55:37.618 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:55:37.618 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.4.asok 2019-12-21T15:55:37.618 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:55:37.619 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.4.asok dump_scrub_reservations 2019-12-21T15:55:37.706 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:55:37.707 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 2, 2019-12-21T15:55:37.707 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 0, 2019-12-21T15:55:37.707 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:55:37.707 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:55:37.715 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:55:37.715 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.4 2019-12-21T15:55:37.715 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2019-12-21T15:55:37.716 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2019-12-21T15:55:37.716 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:55:37.716 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:55:37.716 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:55:37.716 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.4.asok 2019-12-21T15:55:37.716 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:55:37.716 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.4.asok dump_scrub_reservations 2019-12-21T15:55:37.810 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=2 2019-12-21T15:55:37.810 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 2 -gt 4 ']' 2019-12-21T15:55:37.811 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 4 + 2 2019-12-21T15:55:37.811 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=6 2019-12-21T15:55:37.812 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:55:37.813 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.5 2019-12-21T15:55:37.813 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2019-12-21T15:55:37.813 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2019-12-21T15:55:37.813 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:55:37.814 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:55:37.814 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:55:37.814 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.5.asok 2019-12-21T15:55:37.814 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:55:37.814 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.5.asok dump_scrub_reservations 2019-12-21T15:55:37.902 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:55:37.902 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 0, 2019-12-21T15:55:37.902 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 3, 2019-12-21T15:55:37.902 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:55:37.902 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:55:37.910 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:55:37.910 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.5 2019-12-21T15:55:37.911 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2019-12-21T15:55:37.911 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2019-12-21T15:55:37.911 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:55:37.911 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:55:37.911 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:55:37.911 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.5.asok 2019-12-21T15:55:37.912 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:55:37.912 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.5.asok dump_scrub_reservations 2019-12-21T15:55:38.006 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=3 2019-12-21T15:55:38.006 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2019-12-21T15:55:38.007 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 6 + 3 2019-12-21T15:55:38.008 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=9 2019-12-21T15:55:38.008 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:133: TEST_recover_unexpected: '[' 9 = 0 ']' 2019-12-21T15:55:38.008 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:136: TEST_recover_unexpected: '[' 0 -gt 3 ']' 2019-12-21T15:55:38.009 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: expr 590 + 9 2019-12-21T15:55:38.009 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: total=599 2019-12-21T15:55:38.010 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: expr 2 '*' 2 2019-12-21T15:55:38.011 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: sleep 4 2019-12-21T15:55:42.012 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:120: TEST_recover_unexpected: true 2019-12-21T15:55:42.012 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:122: TEST_recover_unexpected: pass=0 2019-12-21T15:55:42.013 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: expr 6 - 1 2019-12-21T15:55:42.015 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: seq 0 5 2019-12-21T15:55:42.016 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:55:42.017 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.0 2019-12-21T15:55:42.017 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2019-12-21T15:55:42.017 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2019-12-21T15:55:42.018 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:55:42.018 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:55:42.018 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:55:42.018 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.0.asok 2019-12-21T15:55:42.019 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:55:42.019 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.0.asok dump_scrub_reservations 2019-12-21T15:55:42.116 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:55:42.116 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 1, 2019-12-21T15:55:42.116 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 2, 2019-12-21T15:55:42.117 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:55:42.117 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:55:42.125 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:55:42.126 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.0 2019-12-21T15:55:42.126 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2019-12-21T15:55:42.126 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2019-12-21T15:55:42.127 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:55:42.127 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:55:42.127 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:55:42.127 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.0.asok 2019-12-21T15:55:42.128 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:55:42.128 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.0.asok dump_scrub_reservations 2019-12-21T15:55:42.224 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=3 2019-12-21T15:55:42.225 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2019-12-21T15:55:42.225 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 0 + 3 2019-12-21T15:55:42.225 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=3 2019-12-21T15:55:42.226 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:55:42.228 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.1 2019-12-21T15:55:42.228 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2019-12-21T15:55:42.228 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2019-12-21T15:55:42.228 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:55:42.229 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:55:42.229 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:55:42.229 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.1.asok 2019-12-21T15:55:42.229 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:55:42.229 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.1.asok dump_scrub_reservations 2019-12-21T15:55:42.325 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:55:42.326 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 1, 2019-12-21T15:55:42.326 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 1, 2019-12-21T15:55:42.326 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:55:42.329 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:55:42.329 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:55:42.329 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.1 2019-12-21T15:55:42.329 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2019-12-21T15:55:42.329 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2019-12-21T15:55:42.330 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:55:42.330 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:55:42.330 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:55:42.330 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.1.asok 2019-12-21T15:55:42.330 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:55:42.330 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.1.asok dump_scrub_reservations 2019-12-21T15:55:42.419 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=2 2019-12-21T15:55:42.420 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 2 -gt 4 ']' 2019-12-21T15:55:42.420 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 3 + 2 2019-12-21T15:55:42.421 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=5 2019-12-21T15:55:42.421 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:55:42.422 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.2 2019-12-21T15:55:42.422 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2019-12-21T15:55:42.423 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2019-12-21T15:55:42.423 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:55:42.423 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:55:42.423 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:55:42.423 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.2.asok 2019-12-21T15:55:42.423 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:55:42.424 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.2.asok dump_scrub_reservations 2019-12-21T15:55:42.511 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:55:42.511 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 0, 2019-12-21T15:55:42.511 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 3, 2019-12-21T15:55:42.511 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:55:42.511 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:55:42.519 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:55:42.519 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.2 2019-12-21T15:55:42.520 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2019-12-21T15:55:42.520 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2019-12-21T15:55:42.520 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:55:42.520 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:55:42.520 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:55:42.520 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.2.asok 2019-12-21T15:55:42.520 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:55:42.521 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.2.asok dump_scrub_reservations 2019-12-21T15:55:42.615 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=3 2019-12-21T15:55:42.615 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2019-12-21T15:55:42.616 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 5 + 3 2019-12-21T15:55:42.616 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=8 2019-12-21T15:55:42.617 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:55:42.618 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.3 2019-12-21T15:55:42.618 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2019-12-21T15:55:42.618 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2019-12-21T15:55:42.619 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:55:42.619 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:55:42.619 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:55:42.619 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.3.asok 2019-12-21T15:55:42.619 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:55:42.619 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.3.asok dump_scrub_reservations 2019-12-21T15:55:42.708 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:55:42.708 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 1, 2019-12-21T15:55:42.708 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 0, 2019-12-21T15:55:42.708 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:55:42.709 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:55:42.717 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:55:42.717 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.3 2019-12-21T15:55:42.717 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2019-12-21T15:55:42.717 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2019-12-21T15:55:42.718 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:55:42.718 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:55:42.718 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:55:42.718 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.3.asok 2019-12-21T15:55:42.718 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:55:42.718 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.3.asok dump_scrub_reservations 2019-12-21T15:55:42.812 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=1 2019-12-21T15:55:42.812 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 1 -gt 4 ']' 2019-12-21T15:55:42.815 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 8 + 1 2019-12-21T15:55:42.815 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=9 2019-12-21T15:55:42.816 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:55:42.816 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.4 2019-12-21T15:55:42.816 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2019-12-21T15:55:42.816 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2019-12-21T15:55:42.816 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:55:42.816 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:55:42.817 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:55:42.817 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.4.asok 2019-12-21T15:55:42.817 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:55:42.817 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.4.asok dump_scrub_reservations 2019-12-21T15:55:42.904 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:55:42.904 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 1, 2019-12-21T15:55:42.905 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 0, 2019-12-21T15:55:42.905 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:55:42.905 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:55:42.913 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:55:42.913 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.4 2019-12-21T15:55:42.913 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2019-12-21T15:55:42.914 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2019-12-21T15:55:42.914 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:55:42.914 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:55:42.914 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:55:42.914 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.4.asok 2019-12-21T15:55:42.914 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:55:42.914 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.4.asok dump_scrub_reservations 2019-12-21T15:55:43.009 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=1 2019-12-21T15:55:43.009 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 1 -gt 4 ']' 2019-12-21T15:55:43.009 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 9 + 1 2019-12-21T15:55:43.010 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=10 2019-12-21T15:55:43.010 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:55:43.012 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.5 2019-12-21T15:55:43.012 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2019-12-21T15:55:43.012 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2019-12-21T15:55:43.012 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:55:43.012 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:55:43.012 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:55:43.013 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.5.asok 2019-12-21T15:55:43.013 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:55:43.013 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.5.asok dump_scrub_reservations 2019-12-21T15:55:43.101 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:55:43.101 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 0, 2019-12-21T15:55:43.102 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 2, 2019-12-21T15:55:43.102 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:55:43.102 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:55:43.110 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:55:43.110 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.5 2019-12-21T15:55:43.111 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2019-12-21T15:55:43.111 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2019-12-21T15:55:43.111 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:55:43.111 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:55:43.112 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:55:43.112 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.5.asok 2019-12-21T15:55:43.112 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:55:43.112 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.5.asok dump_scrub_reservations 2019-12-21T15:55:43.206 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=2 2019-12-21T15:55:43.206 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 2 -gt 4 ']' 2019-12-21T15:55:43.207 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 10 + 2 2019-12-21T15:55:43.207 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=12 2019-12-21T15:55:43.208 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:133: TEST_recover_unexpected: '[' 12 = 0 ']' 2019-12-21T15:55:43.208 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:136: TEST_recover_unexpected: '[' 0 -gt 3 ']' 2019-12-21T15:55:43.208 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: expr 599 + 12 2019-12-21T15:55:43.209 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: total=611 2019-12-21T15:55:43.210 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: expr 2 '*' 2 2019-12-21T15:55:43.210 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: sleep 4 2019-12-21T15:55:47.212 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:120: TEST_recover_unexpected: true 2019-12-21T15:55:47.212 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:122: TEST_recover_unexpected: pass=0 2019-12-21T15:55:47.214 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: expr 6 - 1 2019-12-21T15:55:47.215 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: seq 0 5 2019-12-21T15:55:47.216 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:55:47.217 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.0 2019-12-21T15:55:47.217 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2019-12-21T15:55:47.217 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2019-12-21T15:55:47.217 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:55:47.218 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:55:47.218 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:55:47.218 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.0.asok 2019-12-21T15:55:47.218 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:55:47.219 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.0.asok dump_scrub_reservations 2019-12-21T15:55:47.315 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:55:47.316 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 1, 2019-12-21T15:55:47.316 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 2, 2019-12-21T15:55:47.316 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:55:47.316 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:55:47.325 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:55:47.325 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.0 2019-12-21T15:55:47.325 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2019-12-21T15:55:47.325 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2019-12-21T15:55:47.325 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:55:47.325 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:55:47.326 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:55:47.326 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.0.asok 2019-12-21T15:55:47.326 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:55:47.326 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.0.asok dump_scrub_reservations 2019-12-21T15:55:47.420 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=3 2019-12-21T15:55:47.420 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2019-12-21T15:55:47.421 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 0 + 3 2019-12-21T15:55:47.421 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=3 2019-12-21T15:55:47.422 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:55:47.423 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.1 2019-12-21T15:55:47.423 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2019-12-21T15:55:47.423 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2019-12-21T15:55:47.423 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:55:47.424 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:55:47.424 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:55:47.424 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.1.asok 2019-12-21T15:55:47.424 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:55:47.424 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.1.asok dump_scrub_reservations 2019-12-21T15:55:47.512 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:55:47.513 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 1, 2019-12-21T15:55:47.513 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 2, 2019-12-21T15:55:47.513 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:55:47.513 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:55:47.521 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:55:47.521 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.1 2019-12-21T15:55:47.521 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2019-12-21T15:55:47.522 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2019-12-21T15:55:47.522 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:55:47.522 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:55:47.522 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:55:47.522 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.1.asok 2019-12-21T15:55:47.523 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:55:47.523 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.1.asok dump_scrub_reservations 2019-12-21T15:55:47.617 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=3 2019-12-21T15:55:47.618 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2019-12-21T15:55:47.618 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 3 + 3 2019-12-21T15:55:47.619 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=6 2019-12-21T15:55:47.619 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:55:47.620 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.2 2019-12-21T15:55:47.620 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2019-12-21T15:55:47.621 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2019-12-21T15:55:47.621 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:55:47.621 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:55:47.621 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:55:47.621 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.2.asok 2019-12-21T15:55:47.622 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:55:47.622 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.2.asok dump_scrub_reservations 2019-12-21T15:55:47.709 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:55:47.709 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 0, 2019-12-21T15:55:47.709 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 3, 2019-12-21T15:55:47.709 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:55:47.709 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:55:47.718 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:55:47.718 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.2 2019-12-21T15:55:47.718 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2019-12-21T15:55:47.718 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2019-12-21T15:55:47.718 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:55:47.719 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:55:47.719 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:55:47.719 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.2.asok 2019-12-21T15:55:47.719 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:55:47.719 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.2.asok dump_scrub_reservations 2019-12-21T15:55:47.814 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=3 2019-12-21T15:55:47.815 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2019-12-21T15:55:47.815 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 6 + 3 2019-12-21T15:55:47.816 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=9 2019-12-21T15:55:47.816 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:55:47.817 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.3 2019-12-21T15:55:47.817 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2019-12-21T15:55:47.817 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2019-12-21T15:55:47.817 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:55:47.818 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:55:47.818 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:55:47.818 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.3.asok 2019-12-21T15:55:47.818 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:55:47.818 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.3.asok dump_scrub_reservations 2019-12-21T15:55:47.906 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:55:47.906 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 2, 2019-12-21T15:55:47.906 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 0, 2019-12-21T15:55:47.906 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:55:47.907 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:55:47.915 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:55:47.915 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.3 2019-12-21T15:55:47.915 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2019-12-21T15:55:47.915 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2019-12-21T15:55:47.915 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:55:47.915 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:55:47.916 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:55:47.916 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.3.asok 2019-12-21T15:55:47.916 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:55:47.916 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.3.asok dump_scrub_reservations 2019-12-21T15:55:48.011 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=2 2019-12-21T15:55:48.011 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 2 -gt 4 ']' 2019-12-21T15:55:48.012 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 9 + 2 2019-12-21T15:55:48.013 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=11 2019-12-21T15:55:48.013 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:55:48.014 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.4 2019-12-21T15:55:48.014 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2019-12-21T15:55:48.014 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2019-12-21T15:55:48.015 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:55:48.015 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:55:48.015 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:55:48.015 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.4.asok 2019-12-21T15:55:48.015 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:55:48.015 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.4.asok dump_scrub_reservations 2019-12-21T15:55:48.103 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:55:48.103 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 1, 2019-12-21T15:55:48.103 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 0, 2019-12-21T15:55:48.103 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:55:48.103 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:55:48.111 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:55:48.111 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.4 2019-12-21T15:55:48.112 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2019-12-21T15:55:48.112 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2019-12-21T15:55:48.112 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:55:48.112 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:55:48.112 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:55:48.112 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.4.asok 2019-12-21T15:55:48.113 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:55:48.113 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.4.asok dump_scrub_reservations 2019-12-21T15:55:48.206 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=1 2019-12-21T15:55:48.206 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 1 -gt 4 ']' 2019-12-21T15:55:48.207 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 11 + 1 2019-12-21T15:55:48.208 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=12 2019-12-21T15:55:48.208 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:55:48.209 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.5 2019-12-21T15:55:48.209 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2019-12-21T15:55:48.209 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2019-12-21T15:55:48.210 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:55:48.210 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:55:48.210 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:55:48.210 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.5.asok 2019-12-21T15:55:48.210 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:55:48.211 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.5.asok dump_scrub_reservations 2019-12-21T15:55:48.297 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:55:48.298 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 0, 2019-12-21T15:55:48.298 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 3, 2019-12-21T15:55:48.298 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:55:48.298 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:55:48.306 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:55:48.306 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.5 2019-12-21T15:55:48.307 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2019-12-21T15:55:48.307 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2019-12-21T15:55:48.307 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:55:48.307 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:55:48.307 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:55:48.307 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.5.asok 2019-12-21T15:55:48.308 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:55:48.308 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.5.asok dump_scrub_reservations 2019-12-21T15:55:48.402 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=3 2019-12-21T15:55:48.403 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2019-12-21T15:55:48.403 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 12 + 3 2019-12-21T15:55:48.405 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=15 2019-12-21T15:55:48.405 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:133: TEST_recover_unexpected: '[' 15 = 0 ']' 2019-12-21T15:55:48.405 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:136: TEST_recover_unexpected: '[' 0 -gt 3 ']' 2019-12-21T15:55:48.405 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: expr 611 + 15 2019-12-21T15:55:48.406 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: total=626 2019-12-21T15:55:48.406 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: expr 2 '*' 2 2019-12-21T15:55:48.407 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: sleep 4 2019-12-21T15:55:52.409 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:120: TEST_recover_unexpected: true 2019-12-21T15:55:52.409 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:122: TEST_recover_unexpected: pass=0 2019-12-21T15:55:52.410 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: expr 6 - 1 2019-12-21T15:55:52.411 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: seq 0 5 2019-12-21T15:55:52.412 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:55:52.413 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.0 2019-12-21T15:55:52.413 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2019-12-21T15:55:52.413 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2019-12-21T15:55:52.414 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:55:52.414 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:55:52.415 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:55:52.415 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.0.asok 2019-12-21T15:55:52.415 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:55:52.415 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.0.asok dump_scrub_reservations 2019-12-21T15:55:52.512 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:55:52.512 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 1, 2019-12-21T15:55:52.512 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 3, 2019-12-21T15:55:52.512 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:55:52.512 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:55:52.521 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:55:52.521 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.0 2019-12-21T15:55:52.521 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2019-12-21T15:55:52.521 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2019-12-21T15:55:52.522 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:55:52.522 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:55:52.522 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:55:52.522 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.0.asok 2019-12-21T15:55:52.523 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:55:52.523 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.0.asok dump_scrub_reservations 2019-12-21T15:55:52.618 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=4 2019-12-21T15:55:52.618 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2019-12-21T15:55:52.618 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 0 + 4 2019-12-21T15:55:52.619 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=4 2019-12-21T15:55:52.619 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:55:52.620 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.1 2019-12-21T15:55:52.621 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2019-12-21T15:55:52.621 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2019-12-21T15:55:52.621 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:55:52.621 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:55:52.621 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:55:52.621 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.1.asok 2019-12-21T15:55:52.622 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:55:52.622 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.1.asok dump_scrub_reservations 2019-12-21T15:55:52.709 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:55:52.709 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 1, 2019-12-21T15:55:52.709 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 2, 2019-12-21T15:55:52.710 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:55:52.710 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:55:52.720 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:55:52.720 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.1 2019-12-21T15:55:52.721 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2019-12-21T15:55:52.721 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2019-12-21T15:55:52.721 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:55:52.721 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:55:52.721 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:55:52.721 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.1.asok 2019-12-21T15:55:52.721 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:55:52.722 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.1.asok dump_scrub_reservations 2019-12-21T15:55:52.814 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=3 2019-12-21T15:55:52.814 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2019-12-21T15:55:52.815 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 4 + 3 2019-12-21T15:55:52.815 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=7 2019-12-21T15:55:52.816 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:55:52.817 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.2 2019-12-21T15:55:52.817 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2019-12-21T15:55:52.817 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2019-12-21T15:55:52.817 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:55:52.818 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:55:52.818 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:55:52.818 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.2.asok 2019-12-21T15:55:52.818 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:55:52.818 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.2.asok dump_scrub_reservations 2019-12-21T15:55:52.907 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:55:52.907 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 0, 2019-12-21T15:55:52.907 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 2, 2019-12-21T15:55:52.907 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:55:52.908 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:55:52.915 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:55:52.916 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.2 2019-12-21T15:55:52.916 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2019-12-21T15:55:52.916 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2019-12-21T15:55:52.916 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:55:52.916 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:55:52.917 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:55:52.917 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.2.asok 2019-12-21T15:55:52.917 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:55:52.917 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.2.asok dump_scrub_reservations 2019-12-21T15:55:53.011 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=2 2019-12-21T15:55:53.012 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 2 -gt 4 ']' 2019-12-21T15:55:53.012 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 7 + 2 2019-12-21T15:55:53.013 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=9 2019-12-21T15:55:53.013 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:55:53.015 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.3 2019-12-21T15:55:53.015 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2019-12-21T15:55:53.015 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2019-12-21T15:55:53.015 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:55:53.015 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:55:53.016 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:55:53.016 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.3.asok 2019-12-21T15:55:53.016 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:55:53.016 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.3.asok dump_scrub_reservations 2019-12-21T15:55:53.103 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:55:53.104 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 3, 2019-12-21T15:55:53.104 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 0, 2019-12-21T15:55:53.104 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:55:53.104 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:55:53.113 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:55:53.113 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.3 2019-12-21T15:55:53.113 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2019-12-21T15:55:53.113 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2019-12-21T15:55:53.113 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:55:53.114 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:55:53.114 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:55:53.114 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.3.asok 2019-12-21T15:55:53.114 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:55:53.114 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.3.asok dump_scrub_reservations 2019-12-21T15:55:53.207 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=3 2019-12-21T15:55:53.208 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2019-12-21T15:55:53.208 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 9 + 3 2019-12-21T15:55:53.209 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=12 2019-12-21T15:55:53.209 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:55:53.211 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.4 2019-12-21T15:55:53.211 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2019-12-21T15:55:53.211 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2019-12-21T15:55:53.211 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:55:53.211 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:55:53.211 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:55:53.212 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.4.asok 2019-12-21T15:55:53.212 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:55:53.212 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.4.asok dump_scrub_reservations 2019-12-21T15:55:53.300 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:55:53.300 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 0, 2019-12-21T15:55:53.300 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 1, 2019-12-21T15:55:53.300 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:55:53.301 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:55:53.308 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:55:53.308 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.4 2019-12-21T15:55:53.308 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2019-12-21T15:55:53.309 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2019-12-21T15:55:53.311 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:55:53.311 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:55:53.311 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:55:53.312 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.4.asok 2019-12-21T15:55:53.313 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:55:53.313 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.4.asok dump_scrub_reservations 2019-12-21T15:55:53.405 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=1 2019-12-21T15:55:53.405 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 1 -gt 4 ']' 2019-12-21T15:55:53.405 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 12 + 1 2019-12-21T15:55:53.406 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=13 2019-12-21T15:55:53.407 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:55:53.408 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.5 2019-12-21T15:55:53.408 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2019-12-21T15:55:53.408 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2019-12-21T15:55:53.408 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:55:53.408 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:55:53.408 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:55:53.408 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.5.asok 2019-12-21T15:55:53.409 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:55:53.409 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.5.asok dump_scrub_reservations 2019-12-21T15:55:53.497 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:55:53.498 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 0, 2019-12-21T15:55:53.498 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 2, 2019-12-21T15:55:53.498 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:55:53.498 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:55:53.505 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:55:53.506 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.5 2019-12-21T15:55:53.506 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2019-12-21T15:55:53.506 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2019-12-21T15:55:53.506 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:55:53.506 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:55:53.506 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:55:53.507 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.5.asok 2019-12-21T15:55:53.507 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:55:53.507 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.5.asok dump_scrub_reservations 2019-12-21T15:55:53.603 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=2 2019-12-21T15:55:53.603 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 2 -gt 4 ']' 2019-12-21T15:55:53.603 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 13 + 2 2019-12-21T15:55:53.605 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=15 2019-12-21T15:55:53.605 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:133: TEST_recover_unexpected: '[' 15 = 0 ']' 2019-12-21T15:55:53.605 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:136: TEST_recover_unexpected: '[' 0 -gt 3 ']' 2019-12-21T15:55:53.605 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: expr 626 + 15 2019-12-21T15:55:53.606 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: total=641 2019-12-21T15:55:53.606 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: expr 2 '*' 2 2019-12-21T15:55:53.608 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: sleep 4 2019-12-21T15:55:57.609 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:120: TEST_recover_unexpected: true 2019-12-21T15:55:57.609 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:122: TEST_recover_unexpected: pass=0 2019-12-21T15:55:57.610 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: expr 6 - 1 2019-12-21T15:55:57.612 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: seq 0 5 2019-12-21T15:55:57.612 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:55:57.614 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.0 2019-12-21T15:55:57.614 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2019-12-21T15:55:57.614 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2019-12-21T15:55:57.614 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:55:57.615 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:55:57.615 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:55:57.615 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.0.asok 2019-12-21T15:55:57.616 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:55:57.616 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.0.asok dump_scrub_reservations 2019-12-21T15:55:57.712 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:55:57.712 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 1, 2019-12-21T15:55:57.712 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 3, 2019-12-21T15:55:57.712 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:55:57.712 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:55:57.721 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:55:57.722 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.0 2019-12-21T15:55:57.722 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2019-12-21T15:55:57.722 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2019-12-21T15:55:57.722 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:55:57.722 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:55:57.722 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:55:57.722 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.0.asok 2019-12-21T15:55:57.723 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:55:57.723 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.0.asok dump_scrub_reservations 2019-12-21T15:55:57.818 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=4 2019-12-21T15:55:57.818 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2019-12-21T15:55:57.819 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 0 + 4 2019-12-21T15:55:57.819 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=4 2019-12-21T15:55:57.820 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:55:57.822 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.1 2019-12-21T15:55:57.822 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2019-12-21T15:55:57.822 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2019-12-21T15:55:57.822 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:55:57.822 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:55:57.822 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:55:57.822 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.1.asok 2019-12-21T15:55:57.823 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:55:57.823 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.1.asok dump_scrub_reservations 2019-12-21T15:55:57.909 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:55:57.909 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 0, 2019-12-21T15:55:57.910 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 2, 2019-12-21T15:55:57.910 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:55:57.910 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:55:57.918 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:55:57.918 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.1 2019-12-21T15:55:57.918 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2019-12-21T15:55:57.919 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2019-12-21T15:55:57.919 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:55:57.919 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:55:57.919 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:55:57.919 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.1.asok 2019-12-21T15:55:57.919 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:55:57.919 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.1.asok dump_scrub_reservations 2019-12-21T15:55:58.014 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=2 2019-12-21T15:55:58.014 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 2 -gt 4 ']' 2019-12-21T15:55:58.015 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 4 + 2 2019-12-21T15:55:58.015 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=6 2019-12-21T15:55:58.015 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:55:58.017 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.2 2019-12-21T15:55:58.017 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2019-12-21T15:55:58.017 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2019-12-21T15:55:58.017 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:55:58.017 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:55:58.018 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:55:58.018 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.2.asok 2019-12-21T15:55:58.018 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:55:58.018 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.2.asok dump_scrub_reservations 2019-12-21T15:55:58.105 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:55:58.105 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 0, 2019-12-21T15:55:58.105 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 2, 2019-12-21T15:55:58.105 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:55:58.105 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:55:58.113 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:55:58.114 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.2 2019-12-21T15:55:58.114 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2019-12-21T15:55:58.114 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2019-12-21T15:55:58.114 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:55:58.114 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:55:58.114 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:55:58.115 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.2.asok 2019-12-21T15:55:58.115 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:55:58.115 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.2.asok dump_scrub_reservations 2019-12-21T15:55:58.209 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=2 2019-12-21T15:55:58.209 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 2 -gt 4 ']' 2019-12-21T15:55:58.210 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 6 + 2 2019-12-21T15:55:58.210 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=8 2019-12-21T15:55:58.211 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:55:58.212 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.3 2019-12-21T15:55:58.212 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2019-12-21T15:55:58.212 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2019-12-21T15:55:58.212 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:55:58.213 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:55:58.213 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:55:58.213 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.3.asok 2019-12-21T15:55:58.213 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:55:58.213 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.3.asok dump_scrub_reservations 2019-12-21T15:55:58.301 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:55:58.301 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 4, 2019-12-21T15:55:58.302 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 0, 2019-12-21T15:55:58.302 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:55:58.302 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:55:58.310 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:55:58.310 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.3 2019-12-21T15:55:58.310 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2019-12-21T15:55:58.311 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2019-12-21T15:55:58.311 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:55:58.311 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:55:58.311 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:55:58.311 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.3.asok 2019-12-21T15:55:58.312 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:55:58.312 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.3.asok dump_scrub_reservations 2019-12-21T15:55:58.406 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=4 2019-12-21T15:55:58.406 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2019-12-21T15:55:58.407 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 8 + 4 2019-12-21T15:55:58.407 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=12 2019-12-21T15:55:58.407 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:55:58.409 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.4 2019-12-21T15:55:58.409 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2019-12-21T15:55:58.409 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2019-12-21T15:55:58.409 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:55:58.409 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:55:58.409 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:55:58.409 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.4.asok 2019-12-21T15:55:58.410 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:55:58.410 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.4.asok dump_scrub_reservations 2019-12-21T15:55:58.497 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:55:58.498 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 0, 2019-12-21T15:55:58.498 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 2, 2019-12-21T15:55:58.498 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:55:58.498 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:55:58.506 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:55:58.506 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.4 2019-12-21T15:55:58.506 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2019-12-21T15:55:58.506 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2019-12-21T15:55:58.507 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:55:58.507 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:55:58.507 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:55:58.507 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.4.asok 2019-12-21T15:55:58.507 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:55:58.507 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.4.asok dump_scrub_reservations 2019-12-21T15:55:58.602 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=2 2019-12-21T15:55:58.602 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 2 -gt 4 ']' 2019-12-21T15:55:58.602 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 12 + 2 2019-12-21T15:55:58.603 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=14 2019-12-21T15:55:58.603 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:55:58.604 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.5 2019-12-21T15:55:58.604 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2019-12-21T15:55:58.605 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2019-12-21T15:55:58.605 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:55:58.605 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:55:58.605 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:55:58.605 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.5.asok 2019-12-21T15:55:58.606 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:55:58.606 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.5.asok dump_scrub_reservations 2019-12-21T15:55:58.693 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:55:58.694 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 0, 2019-12-21T15:55:58.694 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 1, 2019-12-21T15:55:58.694 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:55:58.694 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:55:58.702 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:55:58.703 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.5 2019-12-21T15:55:58.703 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2019-12-21T15:55:58.703 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2019-12-21T15:55:58.703 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:55:58.703 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:55:58.703 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:55:58.703 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.5.asok 2019-12-21T15:55:58.704 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:55:58.704 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.5.asok dump_scrub_reservations 2019-12-21T15:55:58.798 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=1 2019-12-21T15:55:58.799 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 1 -gt 4 ']' 2019-12-21T15:55:58.799 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 14 + 1 2019-12-21T15:55:58.800 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=15 2019-12-21T15:55:58.800 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:133: TEST_recover_unexpected: '[' 15 = 0 ']' 2019-12-21T15:55:58.800 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:136: TEST_recover_unexpected: '[' 0 -gt 3 ']' 2019-12-21T15:55:58.801 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: expr 641 + 15 2019-12-21T15:55:58.801 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: total=656 2019-12-21T15:55:58.802 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: expr 2 '*' 2 2019-12-21T15:55:58.803 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: sleep 4 2019-12-21T15:56:02.804 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:120: TEST_recover_unexpected: true 2019-12-21T15:56:02.805 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:122: TEST_recover_unexpected: pass=0 2019-12-21T15:56:02.806 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: expr 6 - 1 2019-12-21T15:56:02.807 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: seq 0 5 2019-12-21T15:56:02.808 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:56:02.810 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.0 2019-12-21T15:56:02.810 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2019-12-21T15:56:02.810 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2019-12-21T15:56:02.811 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:56:02.811 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:56:02.811 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:56:02.811 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.0.asok 2019-12-21T15:56:02.812 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:56:02.812 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.0.asok dump_scrub_reservations 2019-12-21T15:56:02.907 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:56:02.908 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 1, 2019-12-21T15:56:02.908 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 2, 2019-12-21T15:56:02.908 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:56:02.908 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:56:02.917 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:56:02.918 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.0 2019-12-21T15:56:02.918 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2019-12-21T15:56:02.918 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2019-12-21T15:56:02.918 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:56:02.919 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:56:02.919 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:56:02.919 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.0.asok 2019-12-21T15:56:02.920 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:56:02.920 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.0.asok dump_scrub_reservations 2019-12-21T15:56:03.013 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=3 2019-12-21T15:56:03.013 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2019-12-21T15:56:03.014 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 0 + 3 2019-12-21T15:56:03.015 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=3 2019-12-21T15:56:03.015 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:56:03.016 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.1 2019-12-21T15:56:03.016 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2019-12-21T15:56:03.016 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2019-12-21T15:56:03.016 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:56:03.017 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:56:03.017 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:56:03.017 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.1.asok 2019-12-21T15:56:03.017 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:56:03.018 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.1.asok dump_scrub_reservations 2019-12-21T15:56:03.104 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:56:03.104 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 0, 2019-12-21T15:56:03.104 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 2, 2019-12-21T15:56:03.105 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:56:03.105 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:56:03.113 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:56:03.113 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.1 2019-12-21T15:56:03.113 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2019-12-21T15:56:03.113 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2019-12-21T15:56:03.113 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:56:03.114 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:56:03.114 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:56:03.114 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.1.asok 2019-12-21T15:56:03.114 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:56:03.114 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.1.asok dump_scrub_reservations 2019-12-21T15:56:03.208 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=2 2019-12-21T15:56:03.209 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 2 -gt 4 ']' 2019-12-21T15:56:03.209 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 3 + 2 2019-12-21T15:56:03.210 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=5 2019-12-21T15:56:03.210 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:56:03.212 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.2 2019-12-21T15:56:03.212 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2019-12-21T15:56:03.212 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2019-12-21T15:56:03.212 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:56:03.212 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:56:03.212 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:56:03.212 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.2.asok 2019-12-21T15:56:03.213 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:56:03.213 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.2.asok dump_scrub_reservations 2019-12-21T15:56:03.299 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:56:03.299 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 0, 2019-12-21T15:56:03.300 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 1, 2019-12-21T15:56:03.300 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:56:03.300 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:56:03.308 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:56:03.308 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.2 2019-12-21T15:56:03.308 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2019-12-21T15:56:03.308 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2019-12-21T15:56:03.309 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:56:03.309 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:56:03.309 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:56:03.309 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.2.asok 2019-12-21T15:56:03.309 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:56:03.309 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.2.asok dump_scrub_reservations 2019-12-21T15:56:03.404 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=1 2019-12-21T15:56:03.405 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 1 -gt 4 ']' 2019-12-21T15:56:03.405 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 5 + 1 2019-12-21T15:56:03.406 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=6 2019-12-21T15:56:03.406 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:56:03.407 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.3 2019-12-21T15:56:03.407 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2019-12-21T15:56:03.408 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2019-12-21T15:56:03.408 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:56:03.408 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:56:03.408 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:56:03.408 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.3.asok 2019-12-21T15:56:03.408 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:56:03.409 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.3.asok dump_scrub_reservations 2019-12-21T15:56:03.497 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:56:03.497 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 3, 2019-12-21T15:56:03.497 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 0, 2019-12-21T15:56:03.497 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:56:03.497 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:56:03.506 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:56:03.506 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.3 2019-12-21T15:56:03.507 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2019-12-21T15:56:03.507 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2019-12-21T15:56:03.507 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:56:03.507 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:56:03.507 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:56:03.507 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.3.asok 2019-12-21T15:56:03.507 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:56:03.508 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.3.asok dump_scrub_reservations 2019-12-21T15:56:03.601 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=3 2019-12-21T15:56:03.602 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2019-12-21T15:56:03.602 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 6 + 3 2019-12-21T15:56:03.603 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=9 2019-12-21T15:56:03.603 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:56:03.604 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.4 2019-12-21T15:56:03.605 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2019-12-21T15:56:03.605 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2019-12-21T15:56:03.605 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:56:03.605 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:56:03.605 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:56:03.606 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.4.asok 2019-12-21T15:56:03.606 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:56:03.606 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.4.asok dump_scrub_reservations 2019-12-21T15:56:03.692 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:56:03.692 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 0, 2019-12-21T15:56:03.692 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 2, 2019-12-21T15:56:03.692 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:56:03.692 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:56:03.700 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:56:03.701 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.4 2019-12-21T15:56:03.701 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2019-12-21T15:56:03.701 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2019-12-21T15:56:03.701 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:56:03.701 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:56:03.701 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:56:03.702 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.4.asok 2019-12-21T15:56:03.702 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:56:03.702 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.4.asok dump_scrub_reservations 2019-12-21T15:56:03.798 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=2 2019-12-21T15:56:03.798 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 2 -gt 4 ']' 2019-12-21T15:56:03.798 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 9 + 2 2019-12-21T15:56:03.799 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=11 2019-12-21T15:56:03.799 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:56:03.801 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.5 2019-12-21T15:56:03.801 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2019-12-21T15:56:03.801 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2019-12-21T15:56:03.801 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:56:03.802 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:56:03.802 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:56:03.802 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.5.asok 2019-12-21T15:56:03.802 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:56:03.802 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.5.asok dump_scrub_reservations 2019-12-21T15:56:03.889 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:56:03.889 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 0, 2019-12-21T15:56:03.889 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 1, 2019-12-21T15:56:03.889 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:56:03.889 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:56:03.897 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:56:03.898 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.5 2019-12-21T15:56:03.898 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2019-12-21T15:56:03.898 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2019-12-21T15:56:03.898 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:56:03.898 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:56:03.899 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:56:03.899 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.5.asok 2019-12-21T15:56:03.899 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:56:03.899 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.5.asok dump_scrub_reservations 2019-12-21T15:56:03.993 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=1 2019-12-21T15:56:03.993 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 1 -gt 4 ']' 2019-12-21T15:56:03.994 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 11 + 1 2019-12-21T15:56:03.996 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=12 2019-12-21T15:56:03.997 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:133: TEST_recover_unexpected: '[' 12 = 0 ']' 2019-12-21T15:56:03.997 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:136: TEST_recover_unexpected: '[' 0 -gt 3 ']' 2019-12-21T15:56:03.997 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: expr 656 + 12 2019-12-21T15:56:03.997 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: total=668 2019-12-21T15:56:03.998 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: expr 2 '*' 2 2019-12-21T15:56:03.998 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: sleep 4 2019-12-21T15:56:07.999 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:120: TEST_recover_unexpected: true 2019-12-21T15:56:07.999 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:122: TEST_recover_unexpected: pass=0 2019-12-21T15:56:08.000 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: expr 6 - 1 2019-12-21T15:56:08.002 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: seq 0 5 2019-12-21T15:56:08.003 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:56:08.004 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.0 2019-12-21T15:56:08.004 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2019-12-21T15:56:08.004 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2019-12-21T15:56:08.005 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:56:08.005 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:56:08.005 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:56:08.005 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.0.asok 2019-12-21T15:56:08.006 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:56:08.006 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.0.asok dump_scrub_reservations 2019-12-21T15:56:08.101 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:56:08.101 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 0, 2019-12-21T15:56:08.102 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 2, 2019-12-21T15:56:08.102 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:56:08.102 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:56:08.111 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:56:08.111 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.0 2019-12-21T15:56:08.111 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2019-12-21T15:56:08.111 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2019-12-21T15:56:08.112 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:56:08.112 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:56:08.112 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:56:08.112 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.0.asok 2019-12-21T15:56:08.112 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:56:08.112 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.0.asok dump_scrub_reservations 2019-12-21T15:56:08.207 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=2 2019-12-21T15:56:08.208 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 2 -gt 4 ']' 2019-12-21T15:56:08.208 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 0 + 2 2019-12-21T15:56:08.209 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=2 2019-12-21T15:56:08.209 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:56:08.210 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.1 2019-12-21T15:56:08.210 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2019-12-21T15:56:08.210 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2019-12-21T15:56:08.211 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:56:08.211 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:56:08.211 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:56:08.211 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.1.asok 2019-12-21T15:56:08.212 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:56:08.212 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.1.asok dump_scrub_reservations 2019-12-21T15:56:08.298 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:56:08.299 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 0, 2019-12-21T15:56:08.299 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 1, 2019-12-21T15:56:08.299 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:56:08.299 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:56:08.307 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:56:08.307 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.1 2019-12-21T15:56:08.307 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2019-12-21T15:56:08.307 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2019-12-21T15:56:08.308 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:56:08.308 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:56:08.308 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:56:08.308 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.1.asok 2019-12-21T15:56:08.308 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:56:08.308 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.1.asok dump_scrub_reservations 2019-12-21T15:56:08.409 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=1 2019-12-21T15:56:08.409 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 1 -gt 4 ']' 2019-12-21T15:56:08.409 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 2 + 1 2019-12-21T15:56:08.409 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=3 2019-12-21T15:56:08.409 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:56:08.409 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.2 2019-12-21T15:56:08.410 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2019-12-21T15:56:08.410 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2019-12-21T15:56:08.410 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:56:08.410 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:56:08.410 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:56:08.410 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.2.asok 2019-12-21T15:56:08.410 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:56:08.411 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.2.asok dump_scrub_reservations 2019-12-21T15:56:08.494 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:56:08.494 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 0, 2019-12-21T15:56:08.494 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 1, 2019-12-21T15:56:08.494 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:56:08.494 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:56:08.502 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:56:08.502 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.2 2019-12-21T15:56:08.503 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2019-12-21T15:56:08.503 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2019-12-21T15:56:08.503 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:56:08.503 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:56:08.503 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:56:08.503 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.2.asok 2019-12-21T15:56:08.504 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:56:08.504 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.2.asok dump_scrub_reservations 2019-12-21T15:56:08.598 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=1 2019-12-21T15:56:08.599 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 1 -gt 4 ']' 2019-12-21T15:56:08.599 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 3 + 1 2019-12-21T15:56:08.600 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=4 2019-12-21T15:56:08.600 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:56:08.601 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.3 2019-12-21T15:56:08.601 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2019-12-21T15:56:08.601 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2019-12-21T15:56:08.602 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:56:08.602 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:56:08.602 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:56:08.602 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.3.asok 2019-12-21T15:56:08.602 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:56:08.603 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.3.asok dump_scrub_reservations 2019-12-21T15:56:08.690 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:56:08.691 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 4, 2019-12-21T15:56:08.691 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 0, 2019-12-21T15:56:08.691 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:56:08.691 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:56:08.700 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:56:08.700 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.3 2019-12-21T15:56:08.701 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2019-12-21T15:56:08.701 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2019-12-21T15:56:08.701 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:56:08.701 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:56:08.701 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:56:08.701 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.3.asok 2019-12-21T15:56:08.702 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:56:08.702 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.3.asok dump_scrub_reservations 2019-12-21T15:56:08.797 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=4 2019-12-21T15:56:08.797 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 4 -gt 4 ']' 2019-12-21T15:56:08.798 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 4 + 4 2019-12-21T15:56:08.798 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=8 2019-12-21T15:56:08.798 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:56:08.799 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.4 2019-12-21T15:56:08.799 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2019-12-21T15:56:08.801 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2019-12-21T15:56:08.801 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:56:08.801 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:56:08.801 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:56:08.801 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.4.asok 2019-12-21T15:56:08.802 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:56:08.802 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.4.asok dump_scrub_reservations 2019-12-21T15:56:08.887 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:56:08.887 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 0, 2019-12-21T15:56:08.888 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 3, 2019-12-21T15:56:08.888 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:56:08.888 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:56:08.896 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:56:08.896 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.4 2019-12-21T15:56:08.896 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2019-12-21T15:56:08.897 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2019-12-21T15:56:08.897 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:56:08.897 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:56:08.897 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:56:08.897 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.4.asok 2019-12-21T15:56:08.897 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:56:08.897 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.4.asok dump_scrub_reservations 2019-12-21T15:56:08.991 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=3 2019-12-21T15:56:08.991 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2019-12-21T15:56:08.992 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 8 + 3 2019-12-21T15:56:08.993 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=11 2019-12-21T15:56:08.993 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:56:08.994 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.5 2019-12-21T15:56:08.994 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2019-12-21T15:56:08.994 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2019-12-21T15:56:08.995 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:56:08.995 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:56:08.995 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:56:08.995 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.5.asok 2019-12-21T15:56:08.995 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:56:08.996 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.5.asok dump_scrub_reservations 2019-12-21T15:56:09.082 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:56:09.082 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 0, 2019-12-21T15:56:09.082 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 1, 2019-12-21T15:56:09.082 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:56:09.083 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:56:09.091 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:56:09.091 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.5 2019-12-21T15:56:09.091 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2019-12-21T15:56:09.091 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2019-12-21T15:56:09.091 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:56:09.091 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:56:09.092 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:56:09.092 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.5.asok 2019-12-21T15:56:09.092 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:56:09.092 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.5.asok dump_scrub_reservations 2019-12-21T15:56:09.186 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=1 2019-12-21T15:56:09.186 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 1 -gt 4 ']' 2019-12-21T15:56:09.186 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 11 + 1 2019-12-21T15:56:09.187 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=12 2019-12-21T15:56:09.187 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:133: TEST_recover_unexpected: '[' 12 = 0 ']' 2019-12-21T15:56:09.188 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:136: TEST_recover_unexpected: '[' 0 -gt 3 ']' 2019-12-21T15:56:09.188 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: expr 668 + 12 2019-12-21T15:56:09.189 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: total=680 2019-12-21T15:56:09.189 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: expr 2 '*' 2 2019-12-21T15:56:09.190 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: sleep 4 2019-12-21T15:56:13.192 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:120: TEST_recover_unexpected: true 2019-12-21T15:56:13.192 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:122: TEST_recover_unexpected: pass=0 2019-12-21T15:56:13.193 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: expr 6 - 1 2019-12-21T15:56:13.194 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: seq 0 5 2019-12-21T15:56:13.195 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:56:13.197 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.0 2019-12-21T15:56:13.197 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2019-12-21T15:56:13.197 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2019-12-21T15:56:13.197 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:56:13.197 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:56:13.197 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:56:13.198 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.0.asok 2019-12-21T15:56:13.198 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:56:13.198 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.0.asok dump_scrub_reservations 2019-12-21T15:56:13.294 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:56:13.295 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 0, 2019-12-21T15:56:13.295 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 2, 2019-12-21T15:56:13.295 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:56:13.295 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:56:13.305 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:56:13.305 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.0 2019-12-21T15:56:13.305 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2019-12-21T15:56:13.305 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2019-12-21T15:56:13.305 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:56:13.306 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:56:13.306 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:56:13.306 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.0.asok 2019-12-21T15:56:13.306 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:56:13.307 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.0.asok dump_scrub_reservations 2019-12-21T15:56:13.402 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=2 2019-12-21T15:56:13.402 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 2 -gt 4 ']' 2019-12-21T15:56:13.403 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 0 + 2 2019-12-21T15:56:13.404 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=2 2019-12-21T15:56:13.404 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:56:13.405 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.1 2019-12-21T15:56:13.405 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2019-12-21T15:56:13.405 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2019-12-21T15:56:13.406 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:56:13.406 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:56:13.406 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:56:13.406 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.1.asok 2019-12-21T15:56:13.406 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:56:13.406 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.1.asok dump_scrub_reservations 2019-12-21T15:56:13.495 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:56:13.495 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 0, 2019-12-21T15:56:13.495 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 0, 2019-12-21T15:56:13.495 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:56:13.495 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:56:13.504 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:56:13.504 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.1 2019-12-21T15:56:13.504 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2019-12-21T15:56:13.504 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2019-12-21T15:56:13.504 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:56:13.504 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:56:13.505 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:56:13.505 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.1.asok 2019-12-21T15:56:13.505 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:56:13.505 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.1.asok dump_scrub_reservations 2019-12-21T15:56:13.600 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=0 2019-12-21T15:56:13.601 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 0 -gt 4 ']' 2019-12-21T15:56:13.601 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 2 + 0 2019-12-21T15:56:13.602 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=2 2019-12-21T15:56:13.602 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:56:13.603 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.2 2019-12-21T15:56:13.603 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2019-12-21T15:56:13.603 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2019-12-21T15:56:13.604 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:56:13.604 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:56:13.604 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:56:13.604 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.2.asok 2019-12-21T15:56:13.605 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:56:13.605 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.2.asok dump_scrub_reservations 2019-12-21T15:56:13.692 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:56:13.692 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 0, 2019-12-21T15:56:13.693 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 1, 2019-12-21T15:56:13.693 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:56:13.693 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:56:13.701 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:56:13.701 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.2 2019-12-21T15:56:13.702 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2019-12-21T15:56:13.702 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2019-12-21T15:56:13.702 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:56:13.702 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:56:13.702 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:56:13.702 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.2.asok 2019-12-21T15:56:13.702 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:56:13.703 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.2.asok dump_scrub_reservations 2019-12-21T15:56:13.796 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=1 2019-12-21T15:56:13.797 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 1 -gt 4 ']' 2019-12-21T15:56:13.797 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 2 + 1 2019-12-21T15:56:13.798 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=3 2019-12-21T15:56:13.798 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:56:13.799 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.3 2019-12-21T15:56:13.799 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2019-12-21T15:56:13.799 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2019-12-21T15:56:13.800 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:56:13.800 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:56:13.800 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:56:13.800 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.3.asok 2019-12-21T15:56:13.800 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:56:13.801 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.3.asok dump_scrub_reservations 2019-12-21T15:56:13.888 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:56:13.888 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 3, 2019-12-21T15:56:13.888 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 0, 2019-12-21T15:56:13.888 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:56:13.888 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:56:13.897 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:56:13.897 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.3 2019-12-21T15:56:13.897 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2019-12-21T15:56:13.897 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2019-12-21T15:56:13.897 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:56:13.897 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:56:13.898 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:56:13.898 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.3.asok 2019-12-21T15:56:13.898 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:56:13.898 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.3.asok dump_scrub_reservations 2019-12-21T15:56:13.992 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=3 2019-12-21T15:56:13.993 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2019-12-21T15:56:13.993 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 3 + 3 2019-12-21T15:56:13.994 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=6 2019-12-21T15:56:13.994 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:56:13.995 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.4 2019-12-21T15:56:13.995 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2019-12-21T15:56:13.996 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2019-12-21T15:56:13.996 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:56:13.996 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:56:13.996 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:56:13.996 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.4.asok 2019-12-21T15:56:13.996 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:56:13.996 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.4.asok dump_scrub_reservations 2019-12-21T15:56:14.083 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:56:14.083 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 0, 2019-12-21T15:56:14.084 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 3, 2019-12-21T15:56:14.084 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:56:14.084 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:56:14.092 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:56:14.092 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.4 2019-12-21T15:56:14.093 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2019-12-21T15:56:14.093 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2019-12-21T15:56:14.093 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:56:14.093 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:56:14.093 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:56:14.093 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.4.asok 2019-12-21T15:56:14.094 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:56:14.094 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.4.asok dump_scrub_reservations 2019-12-21T15:56:14.188 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=3 2019-12-21T15:56:14.188 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 3 -gt 4 ']' 2019-12-21T15:56:14.189 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 6 + 3 2019-12-21T15:56:14.189 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=9 2019-12-21T15:56:14.190 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:56:14.191 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.5 2019-12-21T15:56:14.192 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2019-12-21T15:56:14.192 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2019-12-21T15:56:14.192 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:56:14.192 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:56:14.192 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:56:14.192 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.5.asok 2019-12-21T15:56:14.192 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:56:14.193 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.5.asok dump_scrub_reservations 2019-12-21T15:56:14.279 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:56:14.279 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 0, 2019-12-21T15:56:14.280 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 0, 2019-12-21T15:56:14.280 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:56:14.280 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:56:14.288 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:56:14.288 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.5 2019-12-21T15:56:14.288 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2019-12-21T15:56:14.289 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2019-12-21T15:56:14.289 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:56:14.289 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:56:14.289 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:56:14.289 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.5.asok 2019-12-21T15:56:14.289 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:56:14.289 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.5.asok dump_scrub_reservations 2019-12-21T15:56:14.382 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=0 2019-12-21T15:56:14.382 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 0 -gt 4 ']' 2019-12-21T15:56:14.383 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 9 + 0 2019-12-21T15:56:14.383 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=9 2019-12-21T15:56:14.384 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:133: TEST_recover_unexpected: '[' 9 = 0 ']' 2019-12-21T15:56:14.384 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:136: TEST_recover_unexpected: '[' 0 -gt 3 ']' 2019-12-21T15:56:14.384 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: expr 680 + 9 2019-12-21T15:56:14.385 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: total=689 2019-12-21T15:56:14.385 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: expr 2 '*' 2 2019-12-21T15:56:14.386 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: sleep 4 2019-12-21T15:56:18.388 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:120: TEST_recover_unexpected: true 2019-12-21T15:56:18.388 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:122: TEST_recover_unexpected: pass=0 2019-12-21T15:56:18.389 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: expr 6 - 1 2019-12-21T15:56:18.390 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: seq 0 5 2019-12-21T15:56:18.391 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:56:18.393 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.0 2019-12-21T15:56:18.393 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2019-12-21T15:56:18.393 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2019-12-21T15:56:18.393 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:56:18.393 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:56:18.394 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:56:18.394 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.0.asok 2019-12-21T15:56:18.394 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:56:18.394 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.0.asok dump_scrub_reservations 2019-12-21T15:56:18.491 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:56:18.492 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 0, 2019-12-21T15:56:18.492 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 1, 2019-12-21T15:56:18.492 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:56:18.492 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:56:18.508 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:56:18.508 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.0 2019-12-21T15:56:18.509 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2019-12-21T15:56:18.509 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2019-12-21T15:56:18.509 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:56:18.509 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:56:18.509 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:56:18.509 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.0.asok 2019-12-21T15:56:18.509 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:56:18.510 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.0.asok dump_scrub_reservations 2019-12-21T15:56:18.597 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=1 2019-12-21T15:56:18.597 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 1 -gt 4 ']' 2019-12-21T15:56:18.598 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 0 + 1 2019-12-21T15:56:18.603 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=1 2019-12-21T15:56:18.603 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:56:18.604 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.1 2019-12-21T15:56:18.604 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2019-12-21T15:56:18.604 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2019-12-21T15:56:18.604 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:56:18.604 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:56:18.604 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:56:18.604 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.1.asok 2019-12-21T15:56:18.605 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:56:18.605 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.1.asok dump_scrub_reservations 2019-12-21T15:56:18.689 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:56:18.689 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 0, 2019-12-21T15:56:18.689 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 0, 2019-12-21T15:56:18.689 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:56:18.689 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:56:18.697 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:56:18.698 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.1 2019-12-21T15:56:18.698 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2019-12-21T15:56:18.698 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2019-12-21T15:56:18.698 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:56:18.698 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:56:18.698 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:56:18.698 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.1.asok 2019-12-21T15:56:18.699 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:56:18.699 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.1.asok dump_scrub_reservations 2019-12-21T15:56:18.793 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=0 2019-12-21T15:56:18.793 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 0 -gt 4 ']' 2019-12-21T15:56:18.793 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 1 + 0 2019-12-21T15:56:18.794 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=1 2019-12-21T15:56:18.794 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:56:18.795 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.2 2019-12-21T15:56:18.796 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2019-12-21T15:56:18.796 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2019-12-21T15:56:18.796 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:56:18.796 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:56:18.796 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:56:18.796 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.2.asok 2019-12-21T15:56:18.797 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:56:18.797 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.2.asok dump_scrub_reservations 2019-12-21T15:56:18.884 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:56:18.884 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 0, 2019-12-21T15:56:18.884 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 1, 2019-12-21T15:56:18.884 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:56:18.885 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:56:18.893 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:56:18.894 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.2 2019-12-21T15:56:18.894 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2019-12-21T15:56:18.894 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2019-12-21T15:56:18.894 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:56:18.894 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:56:18.894 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:56:18.895 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.2.asok 2019-12-21T15:56:18.895 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:56:18.895 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.2.asok dump_scrub_reservations 2019-12-21T15:56:18.988 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=1 2019-12-21T15:56:18.988 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 1 -gt 4 ']' 2019-12-21T15:56:18.989 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 1 + 1 2019-12-21T15:56:18.989 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=2 2019-12-21T15:56:18.990 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:56:18.991 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.3 2019-12-21T15:56:18.991 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2019-12-21T15:56:18.991 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2019-12-21T15:56:18.991 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:56:18.991 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:56:18.991 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:56:18.992 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.3.asok 2019-12-21T15:56:18.992 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:56:18.992 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.3.asok dump_scrub_reservations 2019-12-21T15:56:19.079 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:56:19.080 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 2, 2019-12-21T15:56:19.080 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 0, 2019-12-21T15:56:19.080 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:56:19.080 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:56:19.089 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:56:19.089 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.3 2019-12-21T15:56:19.089 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2019-12-21T15:56:19.090 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2019-12-21T15:56:19.090 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:56:19.090 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:56:19.091 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:56:19.091 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.3.asok 2019-12-21T15:56:19.091 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:56:19.091 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.3.asok dump_scrub_reservations 2019-12-21T15:56:19.184 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=2 2019-12-21T15:56:19.184 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 2 -gt 4 ']' 2019-12-21T15:56:19.184 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 2 + 2 2019-12-21T15:56:19.185 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=4 2019-12-21T15:56:19.185 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:56:19.187 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.4 2019-12-21T15:56:19.187 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2019-12-21T15:56:19.187 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2019-12-21T15:56:19.187 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:56:19.188 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:56:19.188 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:56:19.188 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.4.asok 2019-12-21T15:56:19.188 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:56:19.188 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.4.asok dump_scrub_reservations 2019-12-21T15:56:19.276 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:56:19.276 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 0, 2019-12-21T15:56:19.277 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 2, 2019-12-21T15:56:19.277 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:56:19.277 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:56:19.285 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:56:19.286 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.4 2019-12-21T15:56:19.286 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2019-12-21T15:56:19.286 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2019-12-21T15:56:19.286 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:56:19.286 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:56:19.286 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:56:19.287 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.4.asok 2019-12-21T15:56:19.287 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:56:19.287 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.4.asok dump_scrub_reservations 2019-12-21T15:56:19.383 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=2 2019-12-21T15:56:19.383 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 2 -gt 4 ']' 2019-12-21T15:56:19.383 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 4 + 2 2019-12-21T15:56:19.384 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=6 2019-12-21T15:56:19.384 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:56:19.385 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.5 2019-12-21T15:56:19.386 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2019-12-21T15:56:19.386 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2019-12-21T15:56:19.386 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:56:19.386 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:56:19.386 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:56:19.386 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.5.asok 2019-12-21T15:56:19.386 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:56:19.387 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.5.asok dump_scrub_reservations 2019-12-21T15:56:19.473 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:56:19.473 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 0, 2019-12-21T15:56:19.474 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 0, 2019-12-21T15:56:19.474 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:56:19.474 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:56:19.482 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:56:19.482 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.5 2019-12-21T15:56:19.482 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2019-12-21T15:56:19.482 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2019-12-21T15:56:19.482 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:56:19.483 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:56:19.483 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:56:19.483 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.5.asok 2019-12-21T15:56:19.483 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:56:19.483 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.5.asok dump_scrub_reservations 2019-12-21T15:56:19.579 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=0 2019-12-21T15:56:19.580 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 0 -gt 4 ']' 2019-12-21T15:56:19.580 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 6 + 0 2019-12-21T15:56:19.580 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=6 2019-12-21T15:56:19.580 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:133: TEST_recover_unexpected: '[' 6 = 0 ']' 2019-12-21T15:56:19.580 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:136: TEST_recover_unexpected: '[' 0 -gt 3 ']' 2019-12-21T15:56:19.581 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: expr 689 + 6 2019-12-21T15:56:19.581 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: total=695 2019-12-21T15:56:19.583 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: expr 2 '*' 2 2019-12-21T15:56:19.584 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: sleep 4 2019-12-21T15:56:23.584 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:120: TEST_recover_unexpected: true 2019-12-21T15:56:23.584 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:122: TEST_recover_unexpected: pass=0 2019-12-21T15:56:23.585 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: expr 6 - 1 2019-12-21T15:56:23.586 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: seq 0 5 2019-12-21T15:56:23.587 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:56:23.589 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.0 2019-12-21T15:56:23.589 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2019-12-21T15:56:23.589 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2019-12-21T15:56:23.589 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:56:23.589 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:56:23.589 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:56:23.590 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.0.asok 2019-12-21T15:56:23.590 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:56:23.590 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.0.asok dump_scrub_reservations 2019-12-21T15:56:23.687 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:56:23.687 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 0, 2019-12-21T15:56:23.687 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 1, 2019-12-21T15:56:23.687 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:56:23.688 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:56:23.696 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:56:23.696 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.0 2019-12-21T15:56:23.697 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2019-12-21T15:56:23.697 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2019-12-21T15:56:23.697 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:56:23.697 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:56:23.697 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:56:23.698 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.0.asok 2019-12-21T15:56:23.698 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:56:23.698 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.0.asok dump_scrub_reservations 2019-12-21T15:56:23.792 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=1 2019-12-21T15:56:23.792 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 1 -gt 4 ']' 2019-12-21T15:56:23.793 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 0 + 1 2019-12-21T15:56:23.794 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=1 2019-12-21T15:56:23.794 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:56:23.795 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.1 2019-12-21T15:56:23.795 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2019-12-21T15:56:23.795 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2019-12-21T15:56:23.796 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:56:23.796 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:56:23.796 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:56:23.796 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.1.asok 2019-12-21T15:56:23.796 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:56:23.796 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.1.asok dump_scrub_reservations 2019-12-21T15:56:23.884 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:56:23.884 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 0, 2019-12-21T15:56:23.884 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 0, 2019-12-21T15:56:23.884 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:56:23.885 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:56:23.893 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:56:23.893 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.1 2019-12-21T15:56:23.893 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2019-12-21T15:56:23.893 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2019-12-21T15:56:23.893 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:56:23.893 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:56:23.895 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:56:23.895 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.1.asok 2019-12-21T15:56:23.895 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:56:23.896 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.1.asok dump_scrub_reservations 2019-12-21T15:56:23.989 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=0 2019-12-21T15:56:23.989 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 0 -gt 4 ']' 2019-12-21T15:56:23.990 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 1 + 0 2019-12-21T15:56:23.990 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=1 2019-12-21T15:56:23.991 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:56:23.992 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.2 2019-12-21T15:56:23.992 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2019-12-21T15:56:23.992 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2019-12-21T15:56:23.993 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:56:23.993 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:56:23.993 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:56:23.993 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.2.asok 2019-12-21T15:56:23.993 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:56:23.993 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.2.asok dump_scrub_reservations 2019-12-21T15:56:24.080 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:56:24.081 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 0, 2019-12-21T15:56:24.081 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 1, 2019-12-21T15:56:24.081 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:56:24.081 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:56:24.089 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:56:24.090 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.2 2019-12-21T15:56:24.090 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2019-12-21T15:56:24.090 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2019-12-21T15:56:24.090 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:56:24.090 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:56:24.090 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:56:24.091 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.2.asok 2019-12-21T15:56:24.091 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:56:24.091 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.2.asok dump_scrub_reservations 2019-12-21T15:56:24.185 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=1 2019-12-21T15:56:24.185 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 1 -gt 4 ']' 2019-12-21T15:56:24.186 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 1 + 1 2019-12-21T15:56:24.187 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=2 2019-12-21T15:56:24.187 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:56:24.188 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.3 2019-12-21T15:56:24.189 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2019-12-21T15:56:24.189 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2019-12-21T15:56:24.189 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:56:24.189 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:56:24.189 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:56:24.189 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.3.asok 2019-12-21T15:56:24.190 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:56:24.190 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.3.asok dump_scrub_reservations 2019-12-21T15:56:24.278 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:56:24.278 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 2, 2019-12-21T15:56:24.278 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 0, 2019-12-21T15:56:24.278 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:56:24.279 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:56:24.287 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:56:24.287 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.3 2019-12-21T15:56:24.287 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2019-12-21T15:56:24.287 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2019-12-21T15:56:24.287 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:56:24.288 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:56:24.288 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:56:24.288 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.3.asok 2019-12-21T15:56:24.288 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:56:24.288 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.3.asok dump_scrub_reservations 2019-12-21T15:56:24.382 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=2 2019-12-21T15:56:24.382 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 2 -gt 4 ']' 2019-12-21T15:56:24.383 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 2 + 2 2019-12-21T15:56:24.384 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=4 2019-12-21T15:56:24.384 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:56:24.385 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.4 2019-12-21T15:56:24.385 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2019-12-21T15:56:24.386 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2019-12-21T15:56:24.386 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:56:24.386 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:56:24.386 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:56:24.386 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.4.asok 2019-12-21T15:56:24.386 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:56:24.387 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.4.asok dump_scrub_reservations 2019-12-21T15:56:24.474 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:56:24.474 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 0, 2019-12-21T15:56:24.475 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 2, 2019-12-21T15:56:24.475 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:56:24.475 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:56:24.483 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:56:24.483 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.4 2019-12-21T15:56:24.483 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2019-12-21T15:56:24.484 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2019-12-21T15:56:24.484 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:56:24.484 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:56:24.484 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:56:24.484 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.4.asok 2019-12-21T15:56:24.485 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:56:24.485 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.4.asok dump_scrub_reservations 2019-12-21T15:56:24.579 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=2 2019-12-21T15:56:24.580 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 2 -gt 4 ']' 2019-12-21T15:56:24.580 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 4 + 2 2019-12-21T15:56:24.581 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=6 2019-12-21T15:56:24.581 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:56:24.583 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.5 2019-12-21T15:56:24.583 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2019-12-21T15:56:24.583 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2019-12-21T15:56:24.583 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:56:24.583 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:56:24.583 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:56:24.584 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.5.asok 2019-12-21T15:56:24.584 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:56:24.584 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.5.asok dump_scrub_reservations 2019-12-21T15:56:24.670 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:56:24.671 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 0, 2019-12-21T15:56:24.671 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 0, 2019-12-21T15:56:24.671 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:56:24.671 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:56:24.680 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:56:24.680 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.5 2019-12-21T15:56:24.681 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2019-12-21T15:56:24.681 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2019-12-21T15:56:24.681 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:56:24.681 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:56:24.681 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:56:24.681 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.5.asok 2019-12-21T15:56:24.682 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:56:24.682 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.5.asok dump_scrub_reservations 2019-12-21T15:56:24.775 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=0 2019-12-21T15:56:24.775 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 0 -gt 4 ']' 2019-12-21T15:56:24.775 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 6 + 0 2019-12-21T15:56:24.776 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=6 2019-12-21T15:56:24.776 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:133: TEST_recover_unexpected: '[' 6 = 0 ']' 2019-12-21T15:56:24.776 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:136: TEST_recover_unexpected: '[' 0 -gt 3 ']' 2019-12-21T15:56:24.777 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: expr 695 + 6 2019-12-21T15:56:24.777 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: total=701 2019-12-21T15:56:24.778 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: expr 2 '*' 2 2019-12-21T15:56:24.779 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: sleep 4 2019-12-21T15:56:28.781 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:120: TEST_recover_unexpected: true 2019-12-21T15:56:28.781 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:122: TEST_recover_unexpected: pass=0 2019-12-21T15:56:28.783 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: expr 6 - 1 2019-12-21T15:56:28.783 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: seq 0 5 2019-12-21T15:56:28.784 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:56:28.786 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.0 2019-12-21T15:56:28.786 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2019-12-21T15:56:28.786 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2019-12-21T15:56:28.786 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:56:28.786 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:56:28.787 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:56:28.787 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.0.asok 2019-12-21T15:56:28.787 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:56:28.787 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.0.asok dump_scrub_reservations 2019-12-21T15:56:28.886 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:56:28.886 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 0, 2019-12-21T15:56:28.886 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 0, 2019-12-21T15:56:28.886 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:56:28.887 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:56:28.893 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:56:28.893 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.0 2019-12-21T15:56:28.893 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2019-12-21T15:56:28.893 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2019-12-21T15:56:28.893 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:56:28.894 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:56:28.894 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:56:28.894 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.0.asok 2019-12-21T15:56:28.894 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:56:28.894 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.0.asok dump_scrub_reservations 2019-12-21T15:56:28.989 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=0 2019-12-21T15:56:28.989 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 0 -gt 4 ']' 2019-12-21T15:56:28.989 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 0 + 0 2019-12-21T15:56:28.990 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=0 2019-12-21T15:56:28.990 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:56:28.991 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.1 2019-12-21T15:56:28.991 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2019-12-21T15:56:28.992 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2019-12-21T15:56:28.992 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:56:28.992 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:56:28.992 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:56:28.992 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.1.asok 2019-12-21T15:56:28.993 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:56:28.993 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.1.asok dump_scrub_reservations 2019-12-21T15:56:29.080 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:56:29.080 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 0, 2019-12-21T15:56:29.080 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 0, 2019-12-21T15:56:29.080 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:56:29.080 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:56:29.088 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:56:29.088 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.1 2019-12-21T15:56:29.089 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2019-12-21T15:56:29.089 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2019-12-21T15:56:29.089 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:56:29.089 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:56:29.089 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:56:29.089 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.1.asok 2019-12-21T15:56:29.090 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:56:29.090 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.1.asok dump_scrub_reservations 2019-12-21T15:56:29.185 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=0 2019-12-21T15:56:29.185 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 0 -gt 4 ']' 2019-12-21T15:56:29.186 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 0 + 0 2019-12-21T15:56:29.187 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=0 2019-12-21T15:56:29.187 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:56:29.188 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.2 2019-12-21T15:56:29.188 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2019-12-21T15:56:29.188 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2019-12-21T15:56:29.189 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:56:29.189 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:56:29.189 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:56:29.189 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.2.asok 2019-12-21T15:56:29.189 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:56:29.189 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.2.asok dump_scrub_reservations 2019-12-21T15:56:29.277 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:56:29.277 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 0, 2019-12-21T15:56:29.278 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 1, 2019-12-21T15:56:29.278 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:56:29.278 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:56:29.286 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:56:29.287 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.2 2019-12-21T15:56:29.287 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2019-12-21T15:56:29.287 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2019-12-21T15:56:29.287 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:56:29.287 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:56:29.288 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:56:29.288 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.2.asok 2019-12-21T15:56:29.288 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:56:29.288 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.2.asok dump_scrub_reservations 2019-12-21T15:56:29.382 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=1 2019-12-21T15:56:29.383 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 1 -gt 4 ']' 2019-12-21T15:56:29.383 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 0 + 1 2019-12-21T15:56:29.384 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=1 2019-12-21T15:56:29.384 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:56:29.386 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.3 2019-12-21T15:56:29.386 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2019-12-21T15:56:29.386 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2019-12-21T15:56:29.386 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:56:29.386 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:56:29.387 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:56:29.387 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.3.asok 2019-12-21T15:56:29.387 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:56:29.387 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.3.asok dump_scrub_reservations 2019-12-21T15:56:29.476 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:56:29.476 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 1, 2019-12-21T15:56:29.476 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 0, 2019-12-21T15:56:29.477 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:56:29.477 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:56:29.485 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:56:29.486 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.3 2019-12-21T15:56:29.486 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2019-12-21T15:56:29.486 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2019-12-21T15:56:29.486 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:56:29.486 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:56:29.486 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:56:29.486 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.3.asok 2019-12-21T15:56:29.487 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:56:29.487 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.3.asok dump_scrub_reservations 2019-12-21T15:56:29.580 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=1 2019-12-21T15:56:29.580 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 1 -gt 4 ']' 2019-12-21T15:56:29.581 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 1 + 1 2019-12-21T15:56:29.581 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=2 2019-12-21T15:56:29.582 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:56:29.583 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.4 2019-12-21T15:56:29.583 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2019-12-21T15:56:29.583 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2019-12-21T15:56:29.584 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:56:29.584 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:56:29.584 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:56:29.584 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.4.asok 2019-12-21T15:56:29.584 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:56:29.584 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.4.asok dump_scrub_reservations 2019-12-21T15:56:29.670 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:56:29.671 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 0, 2019-12-21T15:56:29.671 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 1, 2019-12-21T15:56:29.671 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:56:29.671 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:56:29.680 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:56:29.680 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.4 2019-12-21T15:56:29.680 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2019-12-21T15:56:29.681 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2019-12-21T15:56:29.681 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:56:29.681 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:56:29.681 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:56:29.681 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.4.asok 2019-12-21T15:56:29.681 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:56:29.681 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.4.asok dump_scrub_reservations 2019-12-21T15:56:29.776 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=1 2019-12-21T15:56:29.776 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 1 -gt 4 ']' 2019-12-21T15:56:29.776 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 2 + 1 2019-12-21T15:56:29.777 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=3 2019-12-21T15:56:29.777 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:56:29.779 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.5 2019-12-21T15:56:29.779 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2019-12-21T15:56:29.779 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2019-12-21T15:56:29.779 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:56:29.779 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:56:29.780 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:56:29.780 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.5.asok 2019-12-21T15:56:29.780 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:56:29.780 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.5.asok dump_scrub_reservations 2019-12-21T15:56:29.867 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:56:29.867 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 0, 2019-12-21T15:56:29.867 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 0, 2019-12-21T15:56:29.867 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:56:29.867 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:56:29.878 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:56:29.878 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.5 2019-12-21T15:56:29.878 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2019-12-21T15:56:29.878 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2019-12-21T15:56:29.879 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:56:29.879 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:56:29.879 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:56:29.879 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.5.asok 2019-12-21T15:56:29.879 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:56:29.879 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.5.asok dump_scrub_reservations 2019-12-21T15:56:29.971 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=0 2019-12-21T15:56:29.971 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 0 -gt 4 ']' 2019-12-21T15:56:29.971 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 3 + 0 2019-12-21T15:56:29.973 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=3 2019-12-21T15:56:29.973 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:133: TEST_recover_unexpected: '[' 3 = 0 ']' 2019-12-21T15:56:29.973 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:136: TEST_recover_unexpected: '[' 0 -gt 3 ']' 2019-12-21T15:56:29.973 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: expr 701 + 3 2019-12-21T15:56:29.974 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: total=704 2019-12-21T15:56:29.976 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: expr 2 '*' 2 2019-12-21T15:56:29.976 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: sleep 4 2019-12-21T15:56:33.977 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:120: TEST_recover_unexpected: true 2019-12-21T15:56:33.977 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:122: TEST_recover_unexpected: pass=0 2019-12-21T15:56:33.985 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: expr 6 - 1 2019-12-21T15:56:33.985 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: seq 0 5 2019-12-21T15:56:33.986 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:56:33.986 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.0 2019-12-21T15:56:33.986 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2019-12-21T15:56:33.986 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2019-12-21T15:56:33.986 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:56:33.986 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:56:33.986 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:56:33.987 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.0.asok 2019-12-21T15:56:33.987 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:56:33.987 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.0.asok dump_scrub_reservations 2019-12-21T15:56:34.081 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:56:34.081 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 0, 2019-12-21T15:56:34.081 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 0, 2019-12-21T15:56:34.081 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:56:34.081 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:56:34.089 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:56:34.089 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.0 2019-12-21T15:56:34.090 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2019-12-21T15:56:34.090 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2019-12-21T15:56:34.091 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:56:34.091 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:56:34.091 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:56:34.091 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.0.asok 2019-12-21T15:56:34.091 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:56:34.091 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.0.asok dump_scrub_reservations 2019-12-21T15:56:34.187 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=0 2019-12-21T15:56:34.187 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 0 -gt 4 ']' 2019-12-21T15:56:34.188 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 0 + 0 2019-12-21T15:56:34.189 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=0 2019-12-21T15:56:34.189 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:56:34.190 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.1 2019-12-21T15:56:34.190 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2019-12-21T15:56:34.190 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2019-12-21T15:56:34.191 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:56:34.191 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:56:34.191 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:56:34.191 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.1.asok 2019-12-21T15:56:34.191 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:56:34.191 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.1.asok dump_scrub_reservations 2019-12-21T15:56:34.279 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:56:34.279 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 0, 2019-12-21T15:56:34.279 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 0, 2019-12-21T15:56:34.280 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:56:34.280 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:56:34.287 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:56:34.288 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.1 2019-12-21T15:56:34.288 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2019-12-21T15:56:34.288 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2019-12-21T15:56:34.288 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:56:34.288 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:56:34.288 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:56:34.289 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.1.asok 2019-12-21T15:56:34.289 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:56:34.289 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.1.asok dump_scrub_reservations 2019-12-21T15:56:34.383 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=0 2019-12-21T15:56:34.384 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 0 -gt 4 ']' 2019-12-21T15:56:34.384 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 0 + 0 2019-12-21T15:56:34.385 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=0 2019-12-21T15:56:34.385 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:56:34.386 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.2 2019-12-21T15:56:34.386 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2019-12-21T15:56:34.386 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2019-12-21T15:56:34.386 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:56:34.387 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:56:34.387 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:56:34.387 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.2.asok 2019-12-21T15:56:34.387 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:56:34.388 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.2.asok dump_scrub_reservations 2019-12-21T15:56:34.475 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:56:34.475 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 0, 2019-12-21T15:56:34.475 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 0, 2019-12-21T15:56:34.475 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:56:34.475 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:56:34.483 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:56:34.484 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.2 2019-12-21T15:56:34.484 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2019-12-21T15:56:34.484 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2019-12-21T15:56:34.484 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:56:34.485 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:56:34.485 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:56:34.485 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.2.asok 2019-12-21T15:56:34.485 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:56:34.485 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.2.asok dump_scrub_reservations 2019-12-21T15:56:34.579 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=0 2019-12-21T15:56:34.579 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 0 -gt 4 ']' 2019-12-21T15:56:34.579 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 0 + 0 2019-12-21T15:56:34.580 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=0 2019-12-21T15:56:34.580 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:56:34.582 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.3 2019-12-21T15:56:34.582 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2019-12-21T15:56:34.582 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2019-12-21T15:56:34.582 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:56:34.582 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:56:34.582 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:56:34.583 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.3.asok 2019-12-21T15:56:34.583 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:56:34.583 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.3.asok dump_scrub_reservations 2019-12-21T15:56:34.670 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:56:34.670 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 0, 2019-12-21T15:56:34.670 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 0, 2019-12-21T15:56:34.670 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:56:34.670 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:56:34.679 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:56:34.679 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.3 2019-12-21T15:56:34.679 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2019-12-21T15:56:34.679 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2019-12-21T15:56:34.679 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:56:34.680 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:56:34.680 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:56:34.680 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.3.asok 2019-12-21T15:56:34.680 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:56:34.680 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.3.asok dump_scrub_reservations 2019-12-21T15:56:34.775 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=0 2019-12-21T15:56:34.775 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 0 -gt 4 ']' 2019-12-21T15:56:34.776 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 0 + 0 2019-12-21T15:56:34.777 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=0 2019-12-21T15:56:34.777 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:56:34.778 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.4 2019-12-21T15:56:34.778 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2019-12-21T15:56:34.778 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2019-12-21T15:56:34.779 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:56:34.779 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:56:34.779 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:56:34.779 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.4.asok 2019-12-21T15:56:34.779 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:56:34.779 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.4.asok dump_scrub_reservations 2019-12-21T15:56:34.866 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:56:34.867 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 0, 2019-12-21T15:56:34.867 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 0, 2019-12-21T15:56:34.867 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:56:34.867 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:56:34.875 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:56:34.875 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.4 2019-12-21T15:56:34.875 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2019-12-21T15:56:34.876 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2019-12-21T15:56:34.876 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:56:34.876 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:56:34.876 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:56:34.876 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.4.asok 2019-12-21T15:56:34.877 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:56:34.878 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.4.asok dump_scrub_reservations 2019-12-21T15:56:34.970 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=0 2019-12-21T15:56:34.971 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 0 -gt 4 ']' 2019-12-21T15:56:34.971 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 0 + 0 2019-12-21T15:56:34.972 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=0 2019-12-21T15:56:34.972 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:56:34.973 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.5 2019-12-21T15:56:34.973 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2019-12-21T15:56:34.973 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2019-12-21T15:56:34.974 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:56:34.974 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:56:34.974 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:56:34.974 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.5.asok 2019-12-21T15:56:34.975 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:56:34.975 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.5.asok dump_scrub_reservations 2019-12-21T15:56:35.061 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:56:35.062 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 0, 2019-12-21T15:56:35.062 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 0, 2019-12-21T15:56:35.062 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:56:35.062 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:56:35.070 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:56:35.070 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.5 2019-12-21T15:56:35.070 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2019-12-21T15:56:35.070 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2019-12-21T15:56:35.071 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:56:35.071 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:56:35.071 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:56:35.071 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.5.asok 2019-12-21T15:56:35.072 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:56:35.072 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.5.asok dump_scrub_reservations 2019-12-21T15:56:35.167 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=0 2019-12-21T15:56:35.168 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 0 -gt 4 ']' 2019-12-21T15:56:35.168 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 0 + 0 2019-12-21T15:56:35.169 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=0 2019-12-21T15:56:35.169 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:133: TEST_recover_unexpected: '[' 0 = 0 ']' 2019-12-21T15:56:35.169 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:134: TEST_recover_unexpected: expr 0 + 1 2019-12-21T15:56:35.170 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:134: TEST_recover_unexpected: zerocount=1 2019-12-21T15:56:35.170 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:136: TEST_recover_unexpected: '[' 1 -gt 3 ']' 2019-12-21T15:56:35.171 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: expr 704 + 0 2019-12-21T15:56:35.172 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: total=704 2019-12-21T15:56:35.172 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: expr 2 '*' 2 2019-12-21T15:56:35.173 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: sleep 4 2019-12-21T15:56:39.175 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:120: TEST_recover_unexpected: true 2019-12-21T15:56:39.175 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:122: TEST_recover_unexpected: pass=0 2019-12-21T15:56:39.176 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: expr 6 - 1 2019-12-21T15:56:39.177 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: seq 0 5 2019-12-21T15:56:39.178 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:56:39.179 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.0 2019-12-21T15:56:39.179 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2019-12-21T15:56:39.180 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2019-12-21T15:56:39.180 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:56:39.181 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:56:39.181 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:56:39.181 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.0.asok 2019-12-21T15:56:39.181 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:56:39.181 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.0.asok dump_scrub_reservations 2019-12-21T15:56:39.278 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:56:39.278 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 0, 2019-12-21T15:56:39.279 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 0, 2019-12-21T15:56:39.279 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:56:39.279 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:56:39.294 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:56:39.295 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.0 2019-12-21T15:56:39.295 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2019-12-21T15:56:39.295 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2019-12-21T15:56:39.295 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:56:39.295 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:56:39.295 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:56:39.295 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.0.asok 2019-12-21T15:56:39.296 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:56:39.296 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.0.asok dump_scrub_reservations 2019-12-21T15:56:39.384 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=0 2019-12-21T15:56:39.384 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 0 -gt 4 ']' 2019-12-21T15:56:39.385 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 0 + 0 2019-12-21T15:56:39.385 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=0 2019-12-21T15:56:39.386 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:56:39.387 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.1 2019-12-21T15:56:39.387 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2019-12-21T15:56:39.387 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2019-12-21T15:56:39.387 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:56:39.387 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:56:39.387 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:56:39.388 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.1.asok 2019-12-21T15:56:39.388 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:56:39.388 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.1.asok dump_scrub_reservations 2019-12-21T15:56:39.477 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:56:39.477 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 0, 2019-12-21T15:56:39.478 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 0, 2019-12-21T15:56:39.478 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:56:39.478 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:56:39.486 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:56:39.486 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.1 2019-12-21T15:56:39.487 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2019-12-21T15:56:39.487 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2019-12-21T15:56:39.487 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:56:39.487 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:56:39.487 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:56:39.488 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.1.asok 2019-12-21T15:56:39.488 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:56:39.488 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.1.asok dump_scrub_reservations 2019-12-21T15:56:39.580 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=0 2019-12-21T15:56:39.580 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 0 -gt 4 ']' 2019-12-21T15:56:39.581 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 0 + 0 2019-12-21T15:56:39.581 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=0 2019-12-21T15:56:39.581 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:56:39.582 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.2 2019-12-21T15:56:39.583 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2019-12-21T15:56:39.583 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2019-12-21T15:56:39.583 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:56:39.583 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:56:39.583 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:56:39.583 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.2.asok 2019-12-21T15:56:39.584 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:56:39.584 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.2.asok dump_scrub_reservations 2019-12-21T15:56:39.671 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:56:39.671 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 0, 2019-12-21T15:56:39.671 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 0, 2019-12-21T15:56:39.671 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:56:39.672 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:56:39.680 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:56:39.680 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.2 2019-12-21T15:56:39.680 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2019-12-21T15:56:39.681 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2019-12-21T15:56:39.681 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:56:39.681 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:56:39.681 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:56:39.681 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.2.asok 2019-12-21T15:56:39.681 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:56:39.681 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.2.asok dump_scrub_reservations 2019-12-21T15:56:39.774 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=0 2019-12-21T15:56:39.775 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 0 -gt 4 ']' 2019-12-21T15:56:39.775 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 0 + 0 2019-12-21T15:56:39.776 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=0 2019-12-21T15:56:39.776 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:56:39.777 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.3 2019-12-21T15:56:39.777 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2019-12-21T15:56:39.777 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2019-12-21T15:56:39.778 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:56:39.778 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:56:39.778 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:56:39.778 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.3.asok 2019-12-21T15:56:39.779 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:56:39.779 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.3.asok dump_scrub_reservations 2019-12-21T15:56:39.866 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:56:39.866 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 0, 2019-12-21T15:56:39.866 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 0, 2019-12-21T15:56:39.867 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:56:39.867 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:56:39.875 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:56:39.875 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.3 2019-12-21T15:56:39.875 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2019-12-21T15:56:39.875 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2019-12-21T15:56:39.875 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:56:39.876 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:56:39.876 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:56:39.876 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.3.asok 2019-12-21T15:56:39.876 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:56:39.877 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.3.asok dump_scrub_reservations 2019-12-21T15:56:39.970 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=0 2019-12-21T15:56:39.971 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 0 -gt 4 ']' 2019-12-21T15:56:39.971 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 0 + 0 2019-12-21T15:56:39.972 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=0 2019-12-21T15:56:39.972 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:56:39.973 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.4 2019-12-21T15:56:39.973 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2019-12-21T15:56:39.973 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2019-12-21T15:56:39.974 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:56:39.974 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:56:39.974 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:56:39.974 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.4.asok 2019-12-21T15:56:39.975 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:56:39.975 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.4.asok dump_scrub_reservations 2019-12-21T15:56:40.061 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:56:40.062 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 0, 2019-12-21T15:56:40.062 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 0, 2019-12-21T15:56:40.062 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:56:40.062 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:56:40.070 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:56:40.070 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.4 2019-12-21T15:56:40.071 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2019-12-21T15:56:40.071 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2019-12-21T15:56:40.071 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:56:40.071 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:56:40.071 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:56:40.071 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.4.asok 2019-12-21T15:56:40.072 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:56:40.072 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.4.asok dump_scrub_reservations 2019-12-21T15:56:40.166 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=0 2019-12-21T15:56:40.166 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 0 -gt 4 ']' 2019-12-21T15:56:40.166 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 0 + 0 2019-12-21T15:56:40.167 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=0 2019-12-21T15:56:40.167 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:56:40.169 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.5 2019-12-21T15:56:40.169 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2019-12-21T15:56:40.169 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2019-12-21T15:56:40.169 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:56:40.169 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:56:40.170 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:56:40.170 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.5.asok 2019-12-21T15:56:40.170 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:56:40.170 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.5.asok dump_scrub_reservations 2019-12-21T15:56:40.257 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:56:40.258 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 0, 2019-12-21T15:56:40.258 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 0, 2019-12-21T15:56:40.258 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:56:40.258 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:56:40.267 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:56:40.267 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.5 2019-12-21T15:56:40.267 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2019-12-21T15:56:40.267 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2019-12-21T15:56:40.267 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:56:40.268 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:56:40.268 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:56:40.268 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.5.asok 2019-12-21T15:56:40.268 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:56:40.268 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.5.asok dump_scrub_reservations 2019-12-21T15:56:40.362 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=0 2019-12-21T15:56:40.362 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 0 -gt 4 ']' 2019-12-21T15:56:40.363 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 0 + 0 2019-12-21T15:56:40.364 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=0 2019-12-21T15:56:40.364 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:133: TEST_recover_unexpected: '[' 0 = 0 ']' 2019-12-21T15:56:40.365 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:134: TEST_recover_unexpected: expr 1 + 1 2019-12-21T15:56:40.366 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:134: TEST_recover_unexpected: zerocount=2 2019-12-21T15:56:40.366 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:136: TEST_recover_unexpected: '[' 2 -gt 3 ']' 2019-12-21T15:56:40.366 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: expr 704 + 0 2019-12-21T15:56:40.367 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: total=704 2019-12-21T15:56:40.367 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: expr 2 '*' 2 2019-12-21T15:56:40.368 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: sleep 4 2019-12-21T15:56:44.369 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:120: TEST_recover_unexpected: true 2019-12-21T15:56:44.370 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:122: TEST_recover_unexpected: pass=0 2019-12-21T15:56:44.371 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: expr 6 - 1 2019-12-21T15:56:44.372 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: seq 0 5 2019-12-21T15:56:44.373 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:56:44.374 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.0 2019-12-21T15:56:44.374 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2019-12-21T15:56:44.374 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2019-12-21T15:56:44.375 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:56:44.375 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:56:44.375 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:56:44.376 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.0.asok 2019-12-21T15:56:44.376 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:56:44.376 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.0.asok dump_scrub_reservations 2019-12-21T15:56:44.472 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:56:44.472 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 0, 2019-12-21T15:56:44.472 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 0, 2019-12-21T15:56:44.472 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:56:44.472 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:56:44.481 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:56:44.481 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.0 2019-12-21T15:56:44.481 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2019-12-21T15:56:44.482 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2019-12-21T15:56:44.482 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:56:44.482 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:56:44.482 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:56:44.482 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.0.asok 2019-12-21T15:56:44.482 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:56:44.482 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.0.asok dump_scrub_reservations 2019-12-21T15:56:44.583 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=0 2019-12-21T15:56:44.583 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 0 -gt 4 ']' 2019-12-21T15:56:44.583 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 0 + 0 2019-12-21T15:56:44.583 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=0 2019-12-21T15:56:44.584 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:56:44.584 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.1 2019-12-21T15:56:44.584 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2019-12-21T15:56:44.584 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2019-12-21T15:56:44.584 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:56:44.584 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:56:44.584 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:56:44.585 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.1.asok 2019-12-21T15:56:44.585 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:56:44.585 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.1.asok dump_scrub_reservations 2019-12-21T15:56:44.670 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:56:44.670 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 0, 2019-12-21T15:56:44.670 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 0, 2019-12-21T15:56:44.671 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:56:44.671 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:56:44.679 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:56:44.679 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.1 2019-12-21T15:56:44.679 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2019-12-21T15:56:44.679 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2019-12-21T15:56:44.679 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:56:44.679 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:56:44.680 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:56:44.680 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.1.asok 2019-12-21T15:56:44.680 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:56:44.680 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.1.asok dump_scrub_reservations 2019-12-21T15:56:44.774 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=0 2019-12-21T15:56:44.774 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 0 -gt 4 ']' 2019-12-21T15:56:44.775 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 0 + 0 2019-12-21T15:56:44.776 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=0 2019-12-21T15:56:44.776 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:56:44.777 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.2 2019-12-21T15:56:44.777 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2019-12-21T15:56:44.777 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2019-12-21T15:56:44.778 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:56:44.778 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:56:44.778 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:56:44.778 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.2.asok 2019-12-21T15:56:44.778 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:56:44.778 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.2.asok dump_scrub_reservations 2019-12-21T15:56:44.870 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:56:44.871 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 0, 2019-12-21T15:56:44.871 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 0, 2019-12-21T15:56:44.871 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:56:44.871 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:56:44.874 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:56:44.874 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.2 2019-12-21T15:56:44.874 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2019-12-21T15:56:44.874 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2019-12-21T15:56:44.875 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:56:44.875 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:56:44.875 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:56:44.875 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.2.asok 2019-12-21T15:56:44.876 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:56:44.876 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.2.asok dump_scrub_reservations 2019-12-21T15:56:44.969 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=0 2019-12-21T15:56:44.969 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 0 -gt 4 ']' 2019-12-21T15:56:44.970 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 0 + 0 2019-12-21T15:56:44.981 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=0 2019-12-21T15:56:44.982 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:56:44.982 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.3 2019-12-21T15:56:44.982 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2019-12-21T15:56:44.982 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2019-12-21T15:56:44.982 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:56:44.982 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:56:44.982 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:56:44.983 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.3.asok 2019-12-21T15:56:44.983 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:56:44.983 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.3.asok dump_scrub_reservations 2019-12-21T15:56:45.060 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:56:45.061 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 0, 2019-12-21T15:56:45.061 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 0, 2019-12-21T15:56:45.061 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:56:45.061 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:56:45.069 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:56:45.069 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.3 2019-12-21T15:56:45.070 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2019-12-21T15:56:45.070 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2019-12-21T15:56:45.070 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:56:45.070 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:56:45.070 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:56:45.070 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.3.asok 2019-12-21T15:56:45.070 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:56:45.071 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.3.asok dump_scrub_reservations 2019-12-21T15:56:45.166 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=0 2019-12-21T15:56:45.166 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 0 -gt 4 ']' 2019-12-21T15:56:45.167 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 0 + 0 2019-12-21T15:56:45.167 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=0 2019-12-21T15:56:45.167 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:56:45.168 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.4 2019-12-21T15:56:45.168 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2019-12-21T15:56:45.168 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2019-12-21T15:56:45.169 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:56:45.169 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:56:45.169 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:56:45.169 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.4.asok 2019-12-21T15:56:45.169 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:56:45.169 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.4.asok dump_scrub_reservations 2019-12-21T15:56:45.256 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:56:45.256 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 0, 2019-12-21T15:56:45.256 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 0, 2019-12-21T15:56:45.257 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:56:45.257 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:56:45.265 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:56:45.265 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.4 2019-12-21T15:56:45.265 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2019-12-21T15:56:45.265 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2019-12-21T15:56:45.265 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:56:45.265 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:56:45.266 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:56:45.266 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.4.asok 2019-12-21T15:56:45.266 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:56:45.266 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.4.asok dump_scrub_reservations 2019-12-21T15:56:45.360 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=0 2019-12-21T15:56:45.361 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 0 -gt 4 ']' 2019-12-21T15:56:45.361 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 0 + 0 2019-12-21T15:56:45.362 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=0 2019-12-21T15:56:45.362 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:56:45.363 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.5 2019-12-21T15:56:45.363 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2019-12-21T15:56:45.364 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2019-12-21T15:56:45.364 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:56:45.364 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:56:45.364 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:56:45.364 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.5.asok 2019-12-21T15:56:45.365 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:56:45.365 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.5.asok dump_scrub_reservations 2019-12-21T15:56:45.458 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:56:45.458 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 0, 2019-12-21T15:56:45.458 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 0, 2019-12-21T15:56:45.459 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:56:45.459 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:56:45.462 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:56:45.462 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.5 2019-12-21T15:56:45.462 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2019-12-21T15:56:45.462 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2019-12-21T15:56:45.462 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:56:45.463 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:56:45.463 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:56:45.463 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.5.asok 2019-12-21T15:56:45.463 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:56:45.464 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.5.asok dump_scrub_reservations 2019-12-21T15:56:45.558 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=0 2019-12-21T15:56:45.558 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 0 -gt 4 ']' 2019-12-21T15:56:45.559 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 0 + 0 2019-12-21T15:56:45.560 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=0 2019-12-21T15:56:45.560 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:133: TEST_recover_unexpected: '[' 0 = 0 ']' 2019-12-21T15:56:45.560 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:134: TEST_recover_unexpected: expr 2 + 1 2019-12-21T15:56:45.561 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:134: TEST_recover_unexpected: zerocount=3 2019-12-21T15:56:45.561 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:136: TEST_recover_unexpected: '[' 3 -gt 3 ']' 2019-12-21T15:56:45.562 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: expr 704 + 0 2019-12-21T15:56:45.563 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:139: TEST_recover_unexpected: total=704 2019-12-21T15:56:45.564 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: expr 2 '*' 2 2019-12-21T15:56:45.564 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:140: TEST_recover_unexpected: sleep 4 2019-12-21T15:56:49.566 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:120: TEST_recover_unexpected: true 2019-12-21T15:56:49.566 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:122: TEST_recover_unexpected: pass=0 2019-12-21T15:56:49.567 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: expr 6 - 1 2019-12-21T15:56:49.568 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: seq 0 5 2019-12-21T15:56:49.569 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:56:49.571 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.0 2019-12-21T15:56:49.571 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2019-12-21T15:56:49.571 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2019-12-21T15:56:49.571 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:56:49.571 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:56:49.572 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:56:49.572 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.0.asok 2019-12-21T15:56:49.572 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:56:49.573 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.0.asok dump_scrub_reservations 2019-12-21T15:56:49.668 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:56:49.669 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 0, 2019-12-21T15:56:49.669 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 0, 2019-12-21T15:56:49.669 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:56:49.669 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:56:49.678 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:56:49.678 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.0 2019-12-21T15:56:49.678 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2019-12-21T15:56:49.678 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2019-12-21T15:56:49.678 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:56:49.678 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:56:49.679 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:56:49.679 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.0.asok 2019-12-21T15:56:49.679 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:56:49.679 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.0.asok dump_scrub_reservations 2019-12-21T15:56:49.774 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=0 2019-12-21T15:56:49.774 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 0 -gt 4 ']' 2019-12-21T15:56:49.775 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 0 + 0 2019-12-21T15:56:49.775 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=0 2019-12-21T15:56:49.776 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:56:49.777 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.1 2019-12-21T15:56:49.777 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2019-12-21T15:56:49.777 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2019-12-21T15:56:49.778 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:56:49.778 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:56:49.778 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:56:49.778 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.1.asok 2019-12-21T15:56:49.778 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:56:49.778 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.1.asok dump_scrub_reservations 2019-12-21T15:56:49.866 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:56:49.866 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 0, 2019-12-21T15:56:49.866 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 0, 2019-12-21T15:56:49.867 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:56:49.867 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:56:49.875 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:56:49.875 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.1 2019-12-21T15:56:49.875 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2019-12-21T15:56:49.876 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2019-12-21T15:56:49.876 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:56:49.876 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:56:49.876 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:56:49.876 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.1.asok 2019-12-21T15:56:49.877 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:56:49.877 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.1.asok dump_scrub_reservations 2019-12-21T15:56:49.970 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=0 2019-12-21T15:56:49.970 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 0 -gt 4 ']' 2019-12-21T15:56:49.970 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 0 + 0 2019-12-21T15:56:49.971 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=0 2019-12-21T15:56:49.971 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:56:49.973 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.2 2019-12-21T15:56:49.973 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2019-12-21T15:56:49.973 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2019-12-21T15:56:49.973 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:56:49.974 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:56:49.974 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:56:49.974 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.2.asok 2019-12-21T15:56:49.974 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:56:49.974 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.2.asok dump_scrub_reservations 2019-12-21T15:56:50.064 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:56:50.064 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 0, 2019-12-21T15:56:50.064 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 0, 2019-12-21T15:56:50.064 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:56:50.064 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:56:50.071 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:56:50.071 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.2 2019-12-21T15:56:50.072 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2019-12-21T15:56:50.072 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2019-12-21T15:56:50.072 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:56:50.072 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:56:50.072 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:56:50.072 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.2.asok 2019-12-21T15:56:50.072 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:56:50.073 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.2.asok dump_scrub_reservations 2019-12-21T15:56:50.168 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=0 2019-12-21T15:56:50.168 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 0 -gt 4 ']' 2019-12-21T15:56:50.169 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 0 + 0 2019-12-21T15:56:50.169 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=0 2019-12-21T15:56:50.169 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:56:50.171 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.3 2019-12-21T15:56:50.171 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2019-12-21T15:56:50.171 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2019-12-21T15:56:50.171 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:56:50.171 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:56:50.172 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:56:50.172 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.3.asok 2019-12-21T15:56:50.172 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:56:50.172 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.3.asok dump_scrub_reservations 2019-12-21T15:56:50.261 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:56:50.261 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 0, 2019-12-21T15:56:50.261 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 0, 2019-12-21T15:56:50.261 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:56:50.261 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:56:50.272 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:56:50.272 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.3 2019-12-21T15:56:50.272 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2019-12-21T15:56:50.272 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2019-12-21T15:56:50.272 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:56:50.272 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:56:50.273 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:56:50.273 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.3.asok 2019-12-21T15:56:50.273 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:56:50.273 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.3.asok dump_scrub_reservations 2019-12-21T15:56:50.363 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=0 2019-12-21T15:56:50.364 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 0 -gt 4 ']' 2019-12-21T15:56:50.364 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 0 + 0 2019-12-21T15:56:50.365 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=0 2019-12-21T15:56:50.365 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:56:50.366 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.4 2019-12-21T15:56:50.366 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2019-12-21T15:56:50.366 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2019-12-21T15:56:50.367 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:56:50.367 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:56:50.367 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:56:50.367 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.4.asok 2019-12-21T15:56:50.368 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:56:50.368 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.4.asok dump_scrub_reservations 2019-12-21T15:56:50.455 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:56:50.455 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 0, 2019-12-21T15:56:50.455 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 0, 2019-12-21T15:56:50.455 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:56:50.456 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:56:50.464 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:56:50.464 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.4 2019-12-21T15:56:50.464 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.4 2019-12-21T15:56:50.464 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.4 ']' 2019-12-21T15:56:50.464 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:56:50.465 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:56:50.465 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:56:50.465 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.4.asok 2019-12-21T15:56:50.465 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:56:50.465 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.4.asok dump_scrub_reservations 2019-12-21T15:56:50.559 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=0 2019-12-21T15:56:50.559 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 0 -gt 4 ']' 2019-12-21T15:56:50.560 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 0 + 0 2019-12-21T15:56:50.560 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=0 2019-12-21T15:56:50.561 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:123: TEST_recover_unexpected: for o in $(seq 0 $(expr $OSDS - 1)) 2019-12-21T15:56:50.562 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: get_asok_path osd.5 2019-12-21T15:56:50.562 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2019-12-21T15:56:50.562 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2019-12-21T15:56:50.563 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:56:50.563 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:56:50.563 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:56:50.563 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.5.asok 2019-12-21T15:56:50.563 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:56:50.563 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:125: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.5.asok dump_scrub_reservations 2019-12-21T15:56:50.652 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:56:50.652 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_local": 0, 2019-12-21T15:56:50.652 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubs_remote": 0, 2019-12-21T15:56:50.653 INFO:tasks.workunit.client.0.smithi172.stdout: "osd_max_scrubs": 4 2019-12-21T15:56:50.653 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:56:50.659 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: jq '.scrubs_local + .scrubs_remote' 2019-12-21T15:56:50.660 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: get_asok_path osd.5 2019-12-21T15:56:50.660 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2019-12-21T15:56:50.660 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2019-12-21T15:56:50.660 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:56:50.660 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:56:50.661 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:56:50.661 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.43630/ceph-osd.5.asok 2019-12-21T15:56:50.661 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: CEPH_ARGS= 2019-12-21T15:56:50.661 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: ceph daemon /tmp/ceph-asok.43630/ceph-osd.5.asok dump_scrub_reservations 2019-12-21T15:56:50.756 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:126: TEST_recover_unexpected: scrubs=0 2019-12-21T15:56:50.756 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:127: TEST_recover_unexpected: '[' 0 -gt 4 ']' 2019-12-21T15:56:50.757 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: expr 0 + 0 2019-12-21T15:56:50.758 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:131: TEST_recover_unexpected: pass=0 2019-12-21T15:56:50.758 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:133: TEST_recover_unexpected: '[' 0 = 0 ']' 2019-12-21T15:56:50.758 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:134: TEST_recover_unexpected: expr 3 + 1 2019-12-21T15:56:50.759 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:134: TEST_recover_unexpected: zerocount=4 2019-12-21T15:56:50.759 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:136: TEST_recover_unexpected: '[' 4 -gt 3 ']' 2019-12-21T15:56:50.759 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:137: TEST_recover_unexpected: break 2019-12-21T15:56:50.760 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:144: TEST_recover_unexpected: seq 0 5 2019-12-21T15:56:50.760 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:144: TEST_recover_unexpected: for i in $(seq 0 5) 2019-12-21T15:56:50.761 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:146: TEST_recover_unexpected: ceph pg dump pgs 2019-12-21T15:56:50.761 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:146: TEST_recover_unexpected: grep scrubbing 2019-12-21T15:56:50.993 INFO:tasks.workunit.client.0.smithi172.stderr:dumped pgs 2019-12-21T15:56:50.993 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:150: TEST_recover_unexpected: sleep 2 2019-12-21T15:56:52.991 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:144: TEST_recover_unexpected: for i in $(seq 0 5) 2019-12-21T15:56:52.991 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:146: TEST_recover_unexpected: grep scrubbing 2019-12-21T15:56:52.991 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:146: TEST_recover_unexpected: ceph pg dump pgs 2019-12-21T15:56:53.223 INFO:tasks.workunit.client.0.smithi172.stderr:dumped pgs 2019-12-21T15:56:53.231 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:150: TEST_recover_unexpected: sleep 2 2019-12-21T15:56:55.233 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:144: TEST_recover_unexpected: for i in $(seq 0 5) 2019-12-21T15:56:55.233 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:146: TEST_recover_unexpected: ceph pg dump pgs 2019-12-21T15:56:55.233 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:146: TEST_recover_unexpected: grep scrubbing 2019-12-21T15:56:55.468 INFO:tasks.workunit.client.0.smithi172.stderr:dumped pgs 2019-12-21T15:56:55.476 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:150: TEST_recover_unexpected: sleep 2 2019-12-21T15:56:57.477 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:144: TEST_recover_unexpected: for i in $(seq 0 5) 2019-12-21T15:56:57.477 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:146: TEST_recover_unexpected: ceph pg dump pgs 2019-12-21T15:56:57.477 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:146: TEST_recover_unexpected: grep scrubbing 2019-12-21T15:56:57.711 INFO:tasks.workunit.client.0.smithi172.stderr:dumped pgs 2019-12-21T15:56:57.719 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:150: TEST_recover_unexpected: sleep 2 2019-12-21T15:56:59.720 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:144: TEST_recover_unexpected: for i in $(seq 0 5) 2019-12-21T15:56:59.720 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:146: TEST_recover_unexpected: ceph pg dump pgs 2019-12-21T15:56:59.721 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:146: TEST_recover_unexpected: grep scrubbing 2019-12-21T15:56:59.956 INFO:tasks.workunit.client.0.smithi172.stderr:dumped pgs 2019-12-21T15:56:59.964 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:150: TEST_recover_unexpected: sleep 2 2019-12-21T15:57:01.965 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:144: TEST_recover_unexpected: for i in $(seq 0 5) 2019-12-21T15:57:01.965 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:146: TEST_recover_unexpected: ceph pg dump pgs 2019-12-21T15:57:01.966 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:146: TEST_recover_unexpected: grep scrubbing 2019-12-21T15:57:02.196 INFO:tasks.workunit.client.0.smithi172.stderr:dumped pgs 2019-12-21T15:57:02.204 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:150: TEST_recover_unexpected: sleep 2 2019-12-21T15:57:04.205 INFO:tasks.workunit.client.0.smithi172.stdout:704 total reservations seen 2019-12-21T15:57:04.206 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:153: TEST_recover_unexpected: echo 704 total reservations seen 2019-12-21T15:57:04.206 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:158: TEST_recover_unexpected: expr 16 '*' 3 '*' 3 2019-12-21T15:57:04.208 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:158: TEST_recover_unexpected: actual_reservations=144 2019-12-21T15:57:04.208 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:159: TEST_recover_unexpected: '[' 704 -lt 144 ']' 2019-12-21T15:57:04.208 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:164: TEST_recover_unexpected: return 0 2019-12-21T15:57:04.209 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:45: run: teardown td/osd-scrub-dump 2019-12-21T15:57:04.209 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-dump 2019-12-21T15:57:04.209 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2019-12-21T15:57:04.209 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-dump KILL 2019-12-21T15:57:04.210 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2019-12-21T15:57:04.210 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2019-12-21T15:57:04.210 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2019-12-21T15:57:04.210 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2019-12-21T15:57:04.211 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2019-12-21T15:57:04.322 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2019-12-21T15:57:04.322 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2019-12-21T15:57:04.322 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2019-12-21T15:57:04.323 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2019-12-21T15:57:04.324 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2019-12-21T15:57:04.324 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2019-12-21T15:57:04.324 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2019-12-21T15:57:04.326 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local 'pattern=|/usr/lib/systemd/systemd-coredump %P %u %g %s %t %c %h %e' 2019-12-21T15:57:04.327 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' '|' = '|' ']' 2019-12-21T15:57:04.327 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:177: teardown: pattern= 2019-12-21T15:57:04.327 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2019-12-21T15:57:04.327 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname '' 2019-12-21T15:57:04.328 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls . 2019-12-21T15:57:04.328 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2019-12-21T15:57:04.328 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-dump 2019-12-21T15:57:04.430 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2019-12-21T15:57:04.430 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:57:04.430 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:57:04.431 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.43630 2019-12-21T15:57:04.432 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2019-12-21T15:57:04.432 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2019-12-21T15:57:04.432 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2057: main: code=0 2019-12-21T15:57:04.432 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2061: main: teardown td/osd-scrub-dump 0 2019-12-21T15:57:04.433 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-dump 2019-12-21T15:57:04.433 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2019-12-21T15:57:04.433 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-dump KILL 2019-12-21T15:57:04.433 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2019-12-21T15:57:04.433 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2019-12-21T15:57:04.433 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2019-12-21T15:57:04.434 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2019-12-21T15:57:04.434 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2019-12-21T15:57:04.435 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2019-12-21T15:57:04.435 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2019-12-21T15:57:04.435 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2019-12-21T15:57:04.435 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2019-12-21T15:57:04.436 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2019-12-21T15:57:04.436 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2019-12-21T15:57:04.436 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2019-12-21T15:57:04.437 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local 'pattern=|/usr/lib/systemd/systemd-coredump %P %u %g %s %t %c %h %e' 2019-12-21T15:57:04.438 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' '|' = '|' ']' 2019-12-21T15:57:04.438 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:177: teardown: pattern= 2019-12-21T15:57:04.438 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2019-12-21T15:57:04.438 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname '' 2019-12-21T15:57:04.438 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls . 2019-12-21T15:57:04.439 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2019-12-21T15:57:04.439 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-dump 2019-12-21T15:57:04.440 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2019-12-21T15:57:04.440 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:57:04.441 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.43630 2019-12-21T15:57:04.441 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.43630 2019-12-21T15:57:04.442 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2019-12-21T15:57:04.442 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2019-12-21T15:57:04.442 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2062: main: return 0 2019-12-21T15:57:04.442 INFO:teuthology.orchestra.run:Running command with timeout 3600 2019-12-21T15:57:04.443 INFO:teuthology.orchestra.run.smithi172:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2019-12-21T15:57:04.478 INFO:tasks.workunit:Running workunit scrub/osd-scrub-repair.sh... 2019-12-21T15:57:04.479 INFO:teuthology.orchestra.run.smithi172:workunit test scrub/osd-scrub-repair.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=2b368c3feb5f798e3a142f2a972094a812d3d958 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh 2019-12-21T15:57:04.536 INFO:tasks.workunit.client.0.smithi172.stderr:+ source /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh 2019-12-21T15:57:04.536 INFO:tasks.workunit.client.0.smithi172.stderr:++ TIMEOUT=300 2019-12-21T15:57:04.537 INFO:tasks.workunit.client.0.smithi172.stderr:++ WAIT_FOR_CLEAN_TIMEOUT=90 2019-12-21T15:57:04.537 INFO:tasks.workunit.client.0.smithi172.stderr:++ MAX_TIMEOUT=15 2019-12-21T15:57:04.537 INFO:tasks.workunit.client.0.smithi172.stderr:++ PG_NUM=4 2019-12-21T15:57:04.537 INFO:tasks.workunit.client.0.smithi172.stderr:++ TMPDIR=/tmp 2019-12-21T15:57:04.537 INFO:tasks.workunit.client.0.smithi172.stderr:++ CEPH_BUILD_VIRTUALENV=/tmp 2019-12-21T15:57:04.538 INFO:tasks.workunit.client.0.smithi172.stderr:++ TESTDIR=/home/ubuntu/cephtest 2019-12-21T15:57:04.538 INFO:tasks.workunit.client.0.smithi172.stderr:++ type xmlstarlet 2019-12-21T15:57:04.538 INFO:tasks.workunit.client.0.smithi172.stderr:++ XMLSTARLET=xmlstarlet 2019-12-21T15:57:04.538 INFO:tasks.workunit.client.0.smithi172.stderr:+++ uname 2019-12-21T15:57:04.540 INFO:tasks.workunit.client.0.smithi172.stderr:++ '[' Linux = FreeBSD ']' 2019-12-21T15:57:04.540 INFO:tasks.workunit.client.0.smithi172.stderr:++ SED=sed 2019-12-21T15:57:04.540 INFO:tasks.workunit.client.0.smithi172.stderr:++ AWK=awk 2019-12-21T15:57:04.540 INFO:tasks.workunit.client.0.smithi172.stderr:+++ stty -a 2019-12-21T15:57:04.540 INFO:tasks.workunit.client.0.smithi172.stderr:+++ head -1 2019-12-21T15:57:04.544 INFO:tasks.workunit.client.0.smithi172.stderr:+++ sed -e 's/.*columns \([0-9]*\).*/\1/' 2019-12-21T15:57:04.544 INFO:tasks.workunit.client.0.smithi172.stderr:stty: 'standard input': Inappropriate ioctl for device 2019-12-21T15:57:04.544 INFO:tasks.workunit.client.0.smithi172.stderr:++ termwidth= 2019-12-21T15:57:04.544 INFO:tasks.workunit.client.0.smithi172.stderr:++ '[' -n '' -a '' '!=' 0 ']' 2019-12-21T15:57:04.545 INFO:tasks.workunit.client.0.smithi172.stderr:++ DIFFCOLOPTS='-y ' 2019-12-21T15:57:04.545 INFO:tasks.workunit.client.0.smithi172.stderr:++ KERNCORE=kernel.core_pattern 2019-12-21T15:57:04.545 INFO:tasks.workunit.client.0.smithi172.stderr:++ EXTRA_OPTS= 2019-12-21T15:57:04.546 INFO:tasks.workunit.client.0.smithi172.stderr:++ test '' = TESTS 2019-12-21T15:57:04.546 INFO:tasks.workunit.client.0.smithi172.stderr:++ uname 2019-12-21T15:57:04.546 INFO:tasks.workunit.client.0.smithi172.stderr:+ '[' Linux = FreeBSD ']' 2019-12-21T15:57:04.547 INFO:tasks.workunit.client.0.smithi172.stderr:+ use_ec_overwrite=true 2019-12-21T15:57:04.547 INFO:tasks.workunit.client.0.smithi172.stderr:+ getjson=no 2019-12-21T15:57:04.547 INFO:tasks.workunit.client.0.smithi172.stderr:+ jqfilter='def walk(f): 2019-12-21T15:57:04.547 INFO:tasks.workunit.client.0.smithi172.stderr: . as $in 2019-12-21T15:57:04.547 INFO:tasks.workunit.client.0.smithi172.stderr: | if type == "object" then 2019-12-21T15:57:04.547 INFO:tasks.workunit.client.0.smithi172.stderr: reduce keys[] as $key 2019-12-21T15:57:04.547 INFO:tasks.workunit.client.0.smithi172.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2019-12-21T15:57:04.548 INFO:tasks.workunit.client.0.smithi172.stderr: elif type == "array" then map( walk(f) ) | f 2019-12-21T15:57:04.548 INFO:tasks.workunit.client.0.smithi172.stderr: else f 2019-12-21T15:57:04.548 INFO:tasks.workunit.client.0.smithi172.stderr: end; 2019-12-21T15:57:04.548 INFO:tasks.workunit.client.0.smithi172.stderr:walk(if type == "object" then del(.mtime) else . end) 2019-12-21T15:57:04.548 INFO:tasks.workunit.client.0.smithi172.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2019-12-21T15:57:04.549 INFO:tasks.workunit.client.0.smithi172.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2019-12-21T15:57:04.549 INFO:tasks.workunit.client.0.smithi172.stderr:| walk(if type == "object" then del(.version) else . end) 2019-12-21T15:57:04.549 INFO:tasks.workunit.client.0.smithi172.stderr:| walk(if type == "object" then del(.prior_version) else . end)' 2019-12-21T15:57:04.549 INFO:tasks.workunit.client.0.smithi172.stderr:+ sortkeys='import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print json.dumps(ud, sort_keys=True, indent=2)' 2019-12-21T15:57:04.552 INFO:tasks.workunit.client.0.smithi172.stderr:+ main osd-scrub-repair 2019-12-21T15:57:04.552 INFO:tasks.workunit.client.0.smithi172.stderr:+ local dir=td/osd-scrub-repair 2019-12-21T15:57:04.552 INFO:tasks.workunit.client.0.smithi172.stderr:+ shift 2019-12-21T15:57:04.552 INFO:tasks.workunit.client.0.smithi172.stderr:+ shopt -s -o xtrace 2019-12-21T15:57:04.553 INFO:tasks.workunit.client.0.smithi172.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2019-12-21T15:57:04.553 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2050: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2019-12-21T15:57:04.553 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2050: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2019-12-21T15:57:04.553 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2051: main: export PYTHONWARNINGS=ignore 2019-12-21T15:57:04.553 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2051: main: PYTHONWARNINGS=ignore 2019-12-21T15:57:04.554 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2052: main: export CEPH_CONF=/dev/null 2019-12-21T15:57:04.554 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2052: main: CEPH_CONF=/dev/null 2019-12-21T15:57:04.554 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2053: main: unset CEPH_ARGS 2019-12-21T15:57:04.554 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2055: main: local code 2019-12-21T15:57:04.554 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2056: main: run td/osd-scrub-repair 2019-12-21T15:57:04.555 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:51: run: local dir=td/osd-scrub-repair 2019-12-21T15:57:04.555 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:52: run: shift 2019-12-21T15:57:04.555 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:54: run: export CEPH_MON=127.0.0.1:7107 2019-12-21T15:57:04.555 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:54: run: CEPH_MON=127.0.0.1:7107 2019-12-21T15:57:04.555 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:55: run: export CEPH_ARGS 2019-12-21T15:57:04.555 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:56: run: uuidgen 2019-12-21T15:57:04.561 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:56: run: CEPH_ARGS+='--fsid=973e3fc1-e63e-4bde-9db6-84c68fd5e93e --auth-supported=none ' 2019-12-21T15:57:04.561 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:57: run: CEPH_ARGS+='--mon-host=127.0.0.1:7107 ' 2019-12-21T15:57:04.562 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:58: run: CEPH_ARGS+='--osd-skip-data-digest=false ' 2019-12-21T15:57:04.562 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:60: run: export -n CEPH_CLI_TEST_DUP_COMMAND 2019-12-21T15:57:04.562 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:61: run: set 2019-12-21T15:57:04.562 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:61: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2019-12-21T15:57:04.568 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:61: run: local 'funcs=TEST_allow_repair_during_recovery 2019-12-21T15:57:04.568 INFO:tasks.workunit.client.0.smithi172.stderr:TEST_auto_repair_bluestore_basic 2019-12-21T15:57:04.569 INFO:tasks.workunit.client.0.smithi172.stderr:TEST_auto_repair_bluestore_failed 2019-12-21T15:57:04.569 INFO:tasks.workunit.client.0.smithi172.stderr:TEST_auto_repair_bluestore_failed_norecov 2019-12-21T15:57:04.569 INFO:tasks.workunit.client.0.smithi172.stderr:TEST_auto_repair_bluestore_scrub 2019-12-21T15:57:04.569 INFO:tasks.workunit.client.0.smithi172.stderr:TEST_auto_repair_erasure_coded_appends 2019-12-21T15:57:04.569 INFO:tasks.workunit.client.0.smithi172.stderr:TEST_auto_repair_erasure_coded_overwrites 2019-12-21T15:57:04.569 INFO:tasks.workunit.client.0.smithi172.stderr:TEST_corrupt_and_repair_jerasure_appends 2019-12-21T15:57:04.570 INFO:tasks.workunit.client.0.smithi172.stderr:TEST_corrupt_and_repair_jerasure_overwrites 2019-12-21T15:57:04.570 INFO:tasks.workunit.client.0.smithi172.stderr:TEST_corrupt_and_repair_lrc_appends 2019-12-21T15:57:04.570 INFO:tasks.workunit.client.0.smithi172.stderr:TEST_corrupt_and_repair_lrc_overwrites 2019-12-21T15:57:04.570 INFO:tasks.workunit.client.0.smithi172.stderr:TEST_corrupt_and_repair_replicated 2019-12-21T15:57:04.570 INFO:tasks.workunit.client.0.smithi172.stderr:TEST_corrupt_scrub_erasure_appends 2019-12-21T15:57:04.570 INFO:tasks.workunit.client.0.smithi172.stderr:TEST_corrupt_scrub_erasure_overwrites 2019-12-21T15:57:04.570 INFO:tasks.workunit.client.0.smithi172.stderr:TEST_corrupt_scrub_replicated 2019-12-21T15:57:04.571 INFO:tasks.workunit.client.0.smithi172.stderr:TEST_corrupt_snapset_scrub_rep 2019-12-21T15:57:04.571 INFO:tasks.workunit.client.0.smithi172.stderr:TEST_list_missing_erasure_coded_appends 2019-12-21T15:57:04.571 INFO:tasks.workunit.client.0.smithi172.stderr:TEST_list_missing_erasure_coded_overwrites 2019-12-21T15:57:04.571 INFO:tasks.workunit.client.0.smithi172.stderr:TEST_periodic_scrub_replicated 2019-12-21T15:57:04.571 INFO:tasks.workunit.client.0.smithi172.stderr:TEST_repair_stats 2019-12-21T15:57:04.572 INFO:tasks.workunit.client.0.smithi172.stderr:TEST_repair_stats_ec 2019-12-21T15:57:04.572 INFO:tasks.workunit.client.0.smithi172.stderr:TEST_request_scrub_priority 2019-12-21T15:57:04.572 INFO:tasks.workunit.client.0.smithi172.stderr:TEST_scrub_warning 2019-12-21T15:57:04.572 INFO:tasks.workunit.client.0.smithi172.stderr:TEST_skip_non_repair_during_recovery 2019-12-21T15:57:04.572 INFO:tasks.workunit.client.0.smithi172.stderr:TEST_unfound_erasure_coded_appends 2019-12-21T15:57:04.572 INFO:tasks.workunit.client.0.smithi172.stderr:TEST_unfound_erasure_coded_overwrites' 2019-12-21T15:57:04.573 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:62: run: for func in $funcs 2019-12-21T15:57:04.573 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: TEST_allow_repair_during_recovery td/osd-scrub-repair 2019-12-21T15:57:04.573 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:114: TEST_allow_repair_during_recovery: local dir=td/osd-scrub-repair 2019-12-21T15:57:04.573 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:115: TEST_allow_repair_during_recovery: local poolname=rbd 2019-12-21T15:57:04.573 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:117: TEST_allow_repair_during_recovery: setup td/osd-scrub-repair 2019-12-21T15:57:04.573 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2019-12-21T15:57:04.573 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2019-12-21T15:57:04.574 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2019-12-21T15:57:04.574 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2019-12-21T15:57:04.574 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2019-12-21T15:57:04.574 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2019-12-21T15:57:04.574 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2019-12-21T15:57:04.574 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2019-12-21T15:57:04.574 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2019-12-21T15:57:04.575 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2019-12-21T15:57:04.575 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2019-12-21T15:57:04.575 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2019-12-21T15:57:04.575 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2019-12-21T15:57:04.575 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2019-12-21T15:57:04.575 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2019-12-21T15:57:04.576 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2019-12-21T15:57:04.576 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2019-12-21T15:57:04.576 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local 'pattern=|/usr/lib/systemd/systemd-coredump %P %u %g %s %t %c %h %e' 2019-12-21T15:57:04.576 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' '|' = '|' ']' 2019-12-21T15:57:04.576 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:177: teardown: pattern= 2019-12-21T15:57:04.577 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2019-12-21T15:57:04.577 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname '' 2019-12-21T15:57:04.577 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls . 2019-12-21T15:57:04.577 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2019-12-21T15:57:04.577 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2019-12-21T15:57:04.577 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2019-12-21T15:57:04.578 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:57:04.578 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.119575 2019-12-21T15:57:04.578 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.119575 2019-12-21T15:57:04.578 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2019-12-21T15:57:04.578 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2019-12-21T15:57:04.578 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2019-12-21T15:57:04.579 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2019-12-21T15:57:04.579 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:57:04.579 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.119575 2019-12-21T15:57:04.579 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.119575 2019-12-21T15:57:04.579 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2019-12-21T15:57:04.579 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2019-12-21T15:57:04.579 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2019-12-21T15:57:04.580 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2019-12-21T15:57:04.580 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2019-12-21T15:57:04.580 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:118: TEST_allow_repair_during_recovery: run_mon td/osd-scrub-repair a --osd_pool_default_size=2 2019-12-21T15:57:04.580 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2019-12-21T15:57:04.580 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2019-12-21T15:57:04.580 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2019-12-21T15:57:04.580 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2019-12-21T15:57:04.581 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2019-12-21T15:57:04.581 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair --osd_pool_default_size=2 2019-12-21T15:57:04.865 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2019-12-21T15:57:04.866 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2019-12-21T15:57:04.866 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2019-12-21T15:57:04.866 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2019-12-21T15:57:04.866 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:57:04.866 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.119575 2019-12-21T15:57:04.867 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.119575/$cluster-$name.asok' 2019-12-21T15:57:04.867 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.119575/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --osd_pool_default_size=2 2019-12-21T15:57:05.049 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: cat 2019-12-21T15:57:05.051 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: get_config mon a fsid 2019-12-21T15:57:05.051 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: get_config: local daemon=mon 2019-12-21T15:57:05.052 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: get_config: local id=a 2019-12-21T15:57:05.052 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local config=fsid 2019-12-21T15:57:05.052 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: jq -r .fsid 2019-12-21T15:57:05.052 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: get_asok_path mon.a 2019-12-21T15:57:05.052 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2019-12-21T15:57:05.053 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2019-12-21T15:57:05.053 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:57:05.053 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:57:05.053 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.119575 2019-12-21T15:57:05.053 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.119575/ceph-mon.a.asok 2019-12-21T15:57:05.053 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: CEPH_ARGS= 2019-12-21T15:57:05.054 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: ceph --format json daemon /tmp/ceph-asok.119575/ceph-mon.a.asok config get fsid 2019-12-21T15:57:05.139 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: get_config mon a mon_host 2019-12-21T15:57:05.139 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: get_config: local daemon=mon 2019-12-21T15:57:05.139 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: get_config: local id=a 2019-12-21T15:57:05.140 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local config=mon_host 2019-12-21T15:57:05.140 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: jq -r .mon_host 2019-12-21T15:57:05.140 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: get_asok_path mon.a 2019-12-21T15:57:05.140 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2019-12-21T15:57:05.140 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2019-12-21T15:57:05.140 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:57:05.141 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:57:05.141 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.119575 2019-12-21T15:57:05.141 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.119575/ceph-mon.a.asok 2019-12-21T15:57:05.141 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: CEPH_ARGS= 2019-12-21T15:57:05.141 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: ceph --format json daemon /tmp/ceph-asok.119575/ceph-mon.a.asok config get mon_host 2019-12-21T15:57:05.226 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:119: TEST_allow_repair_during_recovery: run_mgr td/osd-scrub-repair x 2019-12-21T15:57:05.227 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:551: run_mgr: local dir=td/osd-scrub-repair 2019-12-21T15:57:05.227 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: shift 2019-12-21T15:57:05.227 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local id=x 2019-12-21T15:57:05.227 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2019-12-21T15:57:05.227 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local data=td/osd-scrub-repair/x 2019-12-21T15:57:05.227 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:558: run_mgr: get_asok_path 2019-12-21T15:57:05.228 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2019-12-21T15:57:05.228 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2019-12-21T15:57:05.228 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2019-12-21T15:57:05.228 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:57:05.228 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.119575 2019-12-21T15:57:05.228 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.119575/$cluster-$name.asok' 2019-12-21T15:57:05.229 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:558: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2019-12-21T15:57:05.229 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:558: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.119575/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2019-12-21T15:57:05.852 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:120: TEST_allow_repair_during_recovery: run_osd td/osd-scrub-repair 0 --osd_scrub_during_recovery=false --osd_repair_during_recovery=true --osd_debug_pretend_recovery_active=true 2019-12-21T15:57:05.852 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local dir=td/osd-scrub-repair 2019-12-21T15:57:05.852 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: shift 2019-12-21T15:57:05.852 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local id=0 2019-12-21T15:57:05.853 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2019-12-21T15:57:05.853 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local osd_data=td/osd-scrub-repair/0 2019-12-21T15:57:05.853 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local 'ceph_args=--fsid=973e3fc1-e63e-4bde-9db6-84c68fd5e93e --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2019-12-21T15:57:05.853 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2019-12-21T15:57:05.853 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-journal-size=100' 2019-12-21T15:57:05.854 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2019-12-21T15:57:05.854 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2019-12-21T15:57:05.854 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2019-12-21T15:57:05.854 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --chdir=' 2019-12-21T15:57:05.854 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+= 2019-12-21T15:57:05.854 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2019-12-21T15:57:05.855 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: get_asok_path 2019-12-21T15:57:05.855 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2019-12-21T15:57:05.855 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2019-12-21T15:57:05.855 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2019-12-21T15:57:05.855 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:57:05.855 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.119575 2019-12-21T15:57:05.855 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.119575/$cluster-$name.asok' 2019-12-21T15:57:05.856 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.119575/$cluster-$name.asok' 2019-12-21T15:57:05.856 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --debug-osd=20' 2019-12-21T15:57:05.856 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-ms=1' 2019-12-21T15:57:05.856 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-monc=20' 2019-12-21T15:57:05.857 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2019-12-21T15:57:05.857 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2019-12-21T15:57:05.857 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2019-12-21T15:57:05.857 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2019-12-21T15:57:05.857 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2019-12-21T15:57:05.858 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' ' 2019-12-21T15:57:05.858 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+='--osd_scrub_during_recovery=false --osd_repair_during_recovery=true --osd_debug_pretend_recovery_active=true' 2019-12-21T15:57:05.858 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: mkdir -p td/osd-scrub-repair/0 2019-12-21T15:57:05.859 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: uuidgen 2019-12-21T15:57:05.859 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: local uuid=ee9f2032-bba1-4e90-b9dd-383b0e131ab8 2019-12-21T15:57:05.860 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: echo 'add osd0 ee9f2032-bba1-4e90-b9dd-383b0e131ab8' 2019-12-21T15:57:05.860 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: ceph-authtool --gen-print-key 2019-12-21T15:57:05.860 INFO:tasks.workunit.client.0.smithi172.stdout:add osd0 ee9f2032-bba1-4e90-b9dd-383b0e131ab8 2019-12-21T15:57:05.874 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: OSD_SECRET=AQDRQP5dNxr1MxAAtxQjmKGJg7vMTA4DzhEByA== 2019-12-21T15:57:05.874 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo '{"cephx_secret": "AQDRQP5dNxr1MxAAtxQjmKGJg7vMTA4DzhEByA=="}' 2019-12-21T15:57:05.874 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph osd new ee9f2032-bba1-4e90-b9dd-383b0e131ab8 -i td/osd-scrub-repair/0/new.json 2019-12-21T15:57:06.132 INFO:tasks.workunit.client.0.smithi172.stdout:0 2019-12-21T15:57:06.142 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: rm td/osd-scrub-repair/0/new.json 2019-12-21T15:57:06.143 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-osd -i 0 --fsid=973e3fc1-e63e-4bde-9db6-84c68fd5e93e --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.119575/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd_scrub_during_recovery=false --osd_repair_during_recovery=true --osd_debug_pretend_recovery_active=true --mkfs --key AQDRQP5dNxr1MxAAtxQjmKGJg7vMTA4DzhEByA== --osd-uuid ee9f2032-bba1-4e90-b9dd-383b0e131ab8 2019-12-21T15:57:06.166 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:57:06.163+0000 7fecbcf8aec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T15:57:06.167 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:57:06.165+0000 7fecbcf8aec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T15:57:06.169 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:57:06.166+0000 7fecbcf8aec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T15:57:06.169 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:57:06.167+0000 7fecbcf8aec0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2019-12-21T15:57:06.170 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:57:06.167+0000 7fecbcf8aec0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2019-12-21T15:57:06.170 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:57:06.167+0000 7fecbcf8aec0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2019-12-21T15:57:06.223 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:57:06.221+0000 7fecbcf8aec0 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2019-12-21T15:57:09.269 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2019-12-21T15:57:09.270 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: cat 2019-12-21T15:57:09.270 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: echo adding osd0 key to auth repository 2019-12-21T15:57:09.271 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2019-12-21T15:57:09.271 INFO:tasks.workunit.client.0.smithi172.stdout:adding osd0 key to auth repository 2019-12-21T15:57:09.548 INFO:tasks.workunit.client.0.smithi172.stdout:start osd.0 2019-12-21T15:57:09.549 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo start osd.0 2019-12-21T15:57:09.549 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph osd dump --format=json 2019-12-21T15:57:09.549 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph-osd -i 0 --fsid=973e3fc1-e63e-4bde-9db6-84c68fd5e93e --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.119575/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd_scrub_during_recovery=false --osd_repair_during_recovery=true --osd_debug_pretend_recovery_active=true 2019-12-21T15:57:09.549 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: jq '.flags_set[]' 2019-12-21T15:57:09.549 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: grep -q '"noup"' 2019-12-21T15:57:09.569 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:57:09.567+0000 7f6c61d67ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T15:57:09.582 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:57:09.579+0000 7f6c61d67ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T15:57:09.584 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:57:09.581+0000 7f6c61d67ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T15:57:09.778 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: wait_for_osd up 0 2019-12-21T15:57:09.778 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:914: wait_for_osd: local state=up 2019-12-21T15:57:09.779 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local id=0 2019-12-21T15:57:09.779 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: status=1 2019-12-21T15:57:09.779 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i=0 )) 2019-12-21T15:57:09.779 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i < 300 )) 2019-12-21T15:57:09.779 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: echo 0 2019-12-21T15:57:09.779 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: ceph osd dump 2019-12-21T15:57:09.779 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: grep 'osd.0 up' 2019-12-21T15:57:09.780 INFO:tasks.workunit.client.0.smithi172.stdout:0 2019-12-21T15:57:10.006 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: sleep 1 2019-12-21T15:57:10.104 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:57:10.101+0000 7f6c61d67ec0 -1 Falling back to public interface 2019-12-21T15:57:10.798 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:57:10.796+0000 7f6c61d67ec0 -1 osd.0 0 log_to_monitors {default=true} 2019-12-21T15:57:11.007 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i++ )) 2019-12-21T15:57:11.007 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i < 300 )) 2019-12-21T15:57:11.007 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: echo 1 2019-12-21T15:57:11.008 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: ceph osd dump 2019-12-21T15:57:11.008 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: grep 'osd.0 up' 2019-12-21T15:57:11.008 INFO:tasks.workunit.client.0.smithi172.stdout:1 2019-12-21T15:57:11.269 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: sleep 1 2019-12-21T15:57:12.271 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i++ )) 2019-12-21T15:57:12.271 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i < 300 )) 2019-12-21T15:57:12.271 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: echo 2 2019-12-21T15:57:12.271 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: grep 'osd.0 up' 2019-12-21T15:57:12.272 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: ceph osd dump 2019-12-21T15:57:12.272 INFO:tasks.workunit.client.0.smithi172.stdout:2 2019-12-21T15:57:12.510 INFO:tasks.workunit.client.0.smithi172.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/119948,v1:127.0.0.1:6803/119948] [v2:127.0.0.1:6804/119948,v1:127.0.0.1:6805/119948] exists,up ee9f2032-bba1-4e90-b9dd-383b0e131ab8 2019-12-21T15:57:12.510 INFO:tasks.workunit.client.0.smithi172.stdout:add osd1 e1be02d3-7106-4dc1-95bf-dd477adfeec5 2019-12-21T15:57:12.510 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=0 2019-12-21T15:57:12.511 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: break 2019-12-21T15:57:12.511 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: return 0 2019-12-21T15:57:12.511 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:123: TEST_allow_repair_during_recovery: run_osd td/osd-scrub-repair 1 --osd_scrub_during_recovery=false --osd_repair_during_recovery=true --osd_debug_pretend_recovery_active=true 2019-12-21T15:57:12.511 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local dir=td/osd-scrub-repair 2019-12-21T15:57:12.511 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: shift 2019-12-21T15:57:12.511 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local id=1 2019-12-21T15:57:12.511 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2019-12-21T15:57:12.512 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local osd_data=td/osd-scrub-repair/1 2019-12-21T15:57:12.512 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local 'ceph_args=--fsid=973e3fc1-e63e-4bde-9db6-84c68fd5e93e --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2019-12-21T15:57:12.512 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2019-12-21T15:57:12.512 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-journal-size=100' 2019-12-21T15:57:12.515 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2019-12-21T15:57:12.515 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2019-12-21T15:57:12.515 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2019-12-21T15:57:12.515 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --chdir=' 2019-12-21T15:57:12.516 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+= 2019-12-21T15:57:12.516 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2019-12-21T15:57:12.517 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: get_asok_path 2019-12-21T15:57:12.517 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2019-12-21T15:57:12.517 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2019-12-21T15:57:12.517 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2019-12-21T15:57:12.517 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:57:12.517 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.119575 2019-12-21T15:57:12.517 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.119575/$cluster-$name.asok' 2019-12-21T15:57:12.518 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.119575/$cluster-$name.asok' 2019-12-21T15:57:12.518 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --debug-osd=20' 2019-12-21T15:57:12.518 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-ms=1' 2019-12-21T15:57:12.518 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-monc=20' 2019-12-21T15:57:12.518 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2019-12-21T15:57:12.518 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2019-12-21T15:57:12.518 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2019-12-21T15:57:12.519 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2019-12-21T15:57:12.519 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2019-12-21T15:57:12.519 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' ' 2019-12-21T15:57:12.519 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+='--osd_scrub_during_recovery=false --osd_repair_during_recovery=true --osd_debug_pretend_recovery_active=true' 2019-12-21T15:57:12.519 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: mkdir -p td/osd-scrub-repair/1 2019-12-21T15:57:12.519 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: uuidgen 2019-12-21T15:57:12.519 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: local uuid=e1be02d3-7106-4dc1-95bf-dd477adfeec5 2019-12-21T15:57:12.520 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: echo 'add osd1 e1be02d3-7106-4dc1-95bf-dd477adfeec5' 2019-12-21T15:57:12.520 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: ceph-authtool --gen-print-key 2019-12-21T15:57:12.523 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: OSD_SECRET=AQDYQP5dfB0KHxAAizQnXUudE8qIg805W7xJZA== 2019-12-21T15:57:12.523 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo '{"cephx_secret": "AQDYQP5dfB0KHxAAizQnXUudE8qIg805W7xJZA=="}' 2019-12-21T15:57:12.524 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph osd new e1be02d3-7106-4dc1-95bf-dd477adfeec5 -i td/osd-scrub-repair/1/new.json 2019-12-21T15:57:12.838 INFO:tasks.workunit.client.0.smithi172.stdout:1 2019-12-21T15:57:12.848 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: rm td/osd-scrub-repair/1/new.json 2019-12-21T15:57:12.848 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-osd -i 1 --fsid=973e3fc1-e63e-4bde-9db6-84c68fd5e93e --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.119575/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd_scrub_during_recovery=false --osd_repair_during_recovery=true --osd_debug_pretend_recovery_active=true --mkfs --key AQDYQP5dfB0KHxAAizQnXUudE8qIg805W7xJZA== --osd-uuid e1be02d3-7106-4dc1-95bf-dd477adfeec5 2019-12-21T15:57:12.871 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:57:12.868+0000 7fb76e216ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T15:57:12.872 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:57:12.870+0000 7fb76e216ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T15:57:12.875 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:57:12.872+0000 7fb76e216ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T15:57:12.875 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:57:12.873+0000 7fb76e216ec0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2019-12-21T15:57:12.875 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:57:12.873+0000 7fb76e216ec0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2019-12-21T15:57:12.876 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:57:12.873+0000 7fb76e216ec0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2019-12-21T15:57:12.929 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:57:12.927+0000 7fb76e216ec0 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2019-12-21T15:57:16.005 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2019-12-21T15:57:16.005 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: cat 2019-12-21T15:57:16.006 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: echo adding osd1 key to auth repository 2019-12-21T15:57:16.006 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2019-12-21T15:57:16.006 INFO:tasks.workunit.client.0.smithi172.stdout:adding osd1 key to auth repository 2019-12-21T15:57:16.290 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo start osd.1 2019-12-21T15:57:16.291 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph osd dump --format=json 2019-12-21T15:57:16.291 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: jq '.flags_set[]' 2019-12-21T15:57:16.291 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: grep -q '"noup"' 2019-12-21T15:57:16.291 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph-osd -i 1 --fsid=973e3fc1-e63e-4bde-9db6-84c68fd5e93e --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.119575/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd_scrub_during_recovery=false --osd_repair_during_recovery=true --osd_debug_pretend_recovery_active=true 2019-12-21T15:57:16.291 INFO:tasks.workunit.client.0.smithi172.stdout:start osd.1 2019-12-21T15:57:16.311 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:57:16.309+0000 7f91273c0ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T15:57:16.313 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:57:16.311+0000 7f91273c0ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T15:57:16.314 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:57:16.311+0000 7f91273c0ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T15:57:16.519 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: wait_for_osd up 1 2019-12-21T15:57:16.519 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:914: wait_for_osd: local state=up 2019-12-21T15:57:16.520 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local id=1 2019-12-21T15:57:16.520 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: status=1 2019-12-21T15:57:16.520 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i=0 )) 2019-12-21T15:57:16.520 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i < 300 )) 2019-12-21T15:57:16.520 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: echo 0 2019-12-21T15:57:16.520 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: ceph osd dump 2019-12-21T15:57:16.521 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: grep 'osd.1 up' 2019-12-21T15:57:16.521 INFO:tasks.workunit.client.0.smithi172.stdout:0 2019-12-21T15:57:16.746 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: sleep 1 2019-12-21T15:57:16.829 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:57:16.826+0000 7f91273c0ec0 -1 Falling back to public interface 2019-12-21T15:57:17.577 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:57:17.574+0000 7f91273c0ec0 -1 osd.1 0 log_to_monitors {default=true} 2019-12-21T15:57:17.748 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i++ )) 2019-12-21T15:57:17.748 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i < 300 )) 2019-12-21T15:57:17.748 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: echo 1 2019-12-21T15:57:17.748 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: ceph osd dump 2019-12-21T15:57:17.748 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: grep 'osd.1 up' 2019-12-21T15:57:17.749 INFO:tasks.workunit.client.0.smithi172.stdout:1 2019-12-21T15:57:17.977 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: sleep 1 2019-12-21T15:57:18.979 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i++ )) 2019-12-21T15:57:18.979 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i < 300 )) 2019-12-21T15:57:18.979 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: echo 2 2019-12-21T15:57:18.979 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: ceph osd dump 2019-12-21T15:57:18.979 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: grep 'osd.1 up' 2019-12-21T15:57:18.980 INFO:tasks.workunit.client.0.smithi172.stdout:2 2019-12-21T15:57:19.216 INFO:tasks.workunit.client.0.smithi172.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/120325,v1:127.0.0.1:6811/120325] [v2:127.0.0.1:6812/120325,v1:127.0.0.1:6813/120325] exists,up e1be02d3-7106-4dc1-95bf-dd477adfeec5 2019-12-21T15:57:19.216 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=0 2019-12-21T15:57:19.217 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: break 2019-12-21T15:57:19.217 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: return 0 2019-12-21T15:57:19.217 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:126: TEST_allow_repair_during_recovery: create_rbd_pool 2019-12-21T15:57:19.217 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2019-12-21T15:57:19.437 INFO:tasks.workunit.client.0.smithi172.stderr:pool 'rbd' does not exist 2019-12-21T15:57:19.444 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:534: create_rbd_pool: create_pool rbd 4 2019-12-21T15:57:19.445 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:539: create_pool: ceph osd pool create rbd 4 2019-12-21T15:57:19.841 INFO:tasks.workunit.client.0.smithi172.stderr:pool 'rbd' created 2019-12-21T15:57:19.849 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: sleep 1 2019-12-21T15:57:20.850 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: rbd pool init rbd 2019-12-21T15:57:21.489 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:127: TEST_allow_repair_during_recovery: wait_for_clean 2019-12-21T15:57:21.489 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1548: wait_for_clean: local cmd= 2019-12-21T15:57:21.489 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1549: wait_for_clean: local num_active_clean=-1 2019-12-21T15:57:21.489 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1550: wait_for_clean: local cur_active_clean 2019-12-21T15:57:21.489 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1551: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2019-12-21T15:57:21.490 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1551: wait_for_clean: get_timeout_delays 90 .1 2019-12-21T15:57:21.490 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1494: get_timeout_delays: shopt -q -o xtrace 2019-12-21T15:57:21.491 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1494: get_timeout_delays: echo true 2019-12-21T15:57:21.491 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1494: get_timeout_delays: local trace=true 2019-12-21T15:57:21.491 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1495: get_timeout_delays: true 2019-12-21T15:57:21.491 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1495: get_timeout_delays: shopt -u -o xtrace 2019-12-21T15:57:21.580 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1551: wait_for_clean: local -a delays 2019-12-21T15:57:21.580 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1552: wait_for_clean: local -i loop=0 2019-12-21T15:57:21.581 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1554: wait_for_clean: flush_pg_stats 2019-12-21T15:57:21.581 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1977: flush_pg_stats: local timeout=300 2019-12-21T15:57:21.581 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1979: flush_pg_stats: ceph osd ls 2019-12-21T15:57:21.807 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1979: flush_pg_stats: ids='0 2019-12-21T15:57:21.807 INFO:tasks.workunit.client.0.smithi172.stderr:1' 2019-12-21T15:57:21.807 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1980: flush_pg_stats: seqs= 2019-12-21T15:57:21.807 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: flush_pg_stats: for osd in $ids 2019-12-21T15:57:21.808 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2019-12-21T15:57:21.920 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: seq=21474836483 2019-12-21T15:57:21.920 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: flush_pg_stats: seqs=' 0-21474836483' 2019-12-21T15:57:21.920 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: flush_pg_stats: for osd in $ids 2019-12-21T15:57:21.920 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2019-12-21T15:57:22.030 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: seq=42949672962 2019-12-21T15:57:22.030 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: flush_pg_stats: seqs=' 0-21474836483 1-42949672962' 2019-12-21T15:57:22.030 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: flush_pg_stats: for s in $seqs 2019-12-21T15:57:22.031 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: echo 0-21474836483 2019-12-21T15:57:22.031 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: cut -d - -f 1 2019-12-21T15:57:22.031 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: osd=0 2019-12-21T15:57:22.033 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: echo 0-21474836483 2019-12-21T15:57:22.033 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: cut -d - -f 2 2019-12-21T15:57:22.034 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: seq=21474836483 2019-12-21T15:57:22.034 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1989: flush_pg_stats: echo 'waiting osd.0 seq 21474836483' 2019-12-21T15:57:22.034 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.0 seq 21474836483 2019-12-21T15:57:22.034 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 0 2019-12-21T15:57:22.262 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 21474836482 -lt 21474836483 2019-12-21T15:57:22.262 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1991: flush_pg_stats: sleep 1 2019-12-21T15:57:23.263 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1992: flush_pg_stats: '[' 300 -eq 0 ']' 2019-12-21T15:57:23.264 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 0 2019-12-21T15:57:23.501 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 21474836484 -lt 21474836483 2019-12-21T15:57:23.501 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: flush_pg_stats: for s in $seqs 2019-12-21T15:57:23.502 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: echo 1-42949672962 2019-12-21T15:57:23.502 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: cut -d - -f 1 2019-12-21T15:57:23.503 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: osd=1 2019-12-21T15:57:23.504 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: echo 1-42949672962 2019-12-21T15:57:23.504 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: cut -d - -f 2 2019-12-21T15:57:23.505 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: seq=42949672962 2019-12-21T15:57:23.505 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1989: flush_pg_stats: echo 'waiting osd.1 seq 42949672962' 2019-12-21T15:57:23.505 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.1 seq 42949672962 2019-12-21T15:57:23.506 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 1 2019-12-21T15:57:23.730 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 42949672962 -lt 42949672962 2019-12-21T15:57:23.731 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1555: wait_for_clean: get_num_pgs 2019-12-21T15:57:23.731 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: ceph --format json status 2019-12-21T15:57:23.731 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: jq .pgmap.num_pgs 2019-12-21T15:57:23.999 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1555: wait_for_clean: test 4 == 0 2019-12-21T15:57:23.999 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1559: wait_for_clean: true 2019-12-21T15:57:24.000 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1563: wait_for_clean: get_num_active_clean 2019-12-21T15:57:24.000 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: get_num_active_clean: local expression 2019-12-21T15:57:24.000 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2019-12-21T15:57:24.001 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("stale") | not)' 2019-12-21T15:57:24.001 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: ceph --format json pg dump pgs 2019-12-21T15:57:24.001 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2019-12-21T15:57:24.223 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1563: wait_for_clean: cur_active_clean=4 2019-12-21T15:57:24.224 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1564: wait_for_clean: get_num_pgs 2019-12-21T15:57:24.224 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: ceph --format json status 2019-12-21T15:57:24.224 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: jq .pgmap.num_pgs 2019-12-21T15:57:24.491 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1564: wait_for_clean: test 4 = 4 2019-12-21T15:57:24.491 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1564: wait_for_clean: break 2019-12-21T15:57:24.491 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: return 0 2019-12-21T15:57:24.492 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:129: TEST_allow_repair_during_recovery: add_something td/osd-scrub-repair rbd 2019-12-21T15:57:24.492 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:68: add_something: local dir=td/osd-scrub-repair 2019-12-21T15:57:24.492 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:69: add_something: local poolname=rbd 2019-12-21T15:57:24.492 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:70: add_something: local obj=SOMETHING 2019-12-21T15:57:24.492 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local scrub=noscrub 2019-12-21T15:57:24.492 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: '[' noscrub = noscrub ']' 2019-12-21T15:57:24.492 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:75: add_something: ceph osd set noscrub 2019-12-21T15:57:24.846 INFO:tasks.workunit.client.0.smithi172.stderr:noscrub is set 2019-12-21T15:57:24.857 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: ceph osd set nodeep-scrub 2019-12-21T15:57:25.213 INFO:tasks.workunit.client.0.smithi172.stderr:nodeep-scrub is set 2019-12-21T15:57:25.221 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:82: add_something: local payload=ABCDEF 2019-12-21T15:57:25.222 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:83: add_something: echo ABCDEF 2019-12-21T15:57:25.222 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:84: add_something: rados --pool rbd put SOMETHING td/osd-scrub-repair/ORIGINAL 2019-12-21T15:57:25.305 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:130: TEST_allow_repair_during_recovery: get_not_primary rbd SOMETHING 2019-12-21T15:57:25.305 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1165: get_not_primary: local poolname=rbd 2019-12-21T15:57:25.306 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1166: get_not_primary: local objectname=SOMETHING 2019-12-21T15:57:25.306 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: get_not_primary: get_primary rbd SOMETHING 2019-12-21T15:57:25.306 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_primary: local poolname=rbd 2019-12-21T15:57:25.306 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1133: get_primary: local objectname=SOMETHING 2019-12-21T15:57:25.306 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1135: get_primary: ceph --format json osd map rbd SOMETHING 2019-12-21T15:57:25.306 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1136: get_primary: jq .acting_primary 2019-12-21T15:57:25.537 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: get_not_primary: local primary=1 2019-12-21T15:57:25.537 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1169: get_not_primary: ceph --format json osd map rbd SOMETHING 2019-12-21T15:57:25.537 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1170: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2019-12-21T15:57:25.777 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:130: TEST_allow_repair_during_recovery: corrupt_and_repair_one td/osd-scrub-repair rbd 0 2019-12-21T15:57:25.777 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:232: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2019-12-21T15:57:25.777 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:233: corrupt_and_repair_one: local poolname=rbd 2019-12-21T15:57:25.777 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:234: corrupt_and_repair_one: local osd=0 2019-12-21T15:57:25.778 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:239: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 0 SOMETHING remove 2019-12-21T15:57:25.778 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: objectstore_tool: local dir=td/osd-scrub-repair 2019-12-21T15:57:25.778 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: objectstore_tool: shift 2019-12-21T15:57:25.778 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1235: objectstore_tool: local id=0 2019-12-21T15:57:25.778 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1236: objectstore_tool: shift 2019-12-21T15:57:25.778 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1238: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING remove 2019-12-21T15:57:25.778 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1205: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2019-12-21T15:57:25.779 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nowait: shift 2019-12-21T15:57:25.779 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1207: _objectstore_tool_nowait: local id=0 2019-12-21T15:57:25.779 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1208: _objectstore_tool_nowait: shift 2019-12-21T15:57:25.779 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1210: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2019-12-21T15:57:25.779 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2019-12-21T15:57:25.779 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2019-12-21T15:57:25.779 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2019-12-21T15:57:25.780 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2019-12-21T15:57:25.780 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2019-12-21T15:57:25.883 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2019-12-21T15:57:25.883 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1212: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING remove 2019-12-21T15:57:25.883 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1193: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2019-12-21T15:57:25.884 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1194: _objectstore_tool_nodown: shift 2019-12-21T15:57:25.884 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1195: _objectstore_tool_nodown: local id=0 2019-12-21T15:57:25.884 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: _objectstore_tool_nodown: shift 2019-12-21T15:57:25.884 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2019-12-21T15:57:25.884 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING remove 2019-12-21T15:57:26.477 INFO:tasks.workunit.client.0.smithi172.stdout:remove #1:eb822e21:::SOMETHING:head# 2019-12-21T15:57:26.996 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1213: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2019-12-21T15:57:26.997 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:842: activate_osd: local dir=td/osd-scrub-repair 2019-12-21T15:57:26.997 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: shift 2019-12-21T15:57:26.997 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: local id=0 2019-12-21T15:57:26.997 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: shift 2019-12-21T15:57:26.997 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local osd_data=td/osd-scrub-repair/0 2019-12-21T15:57:26.997 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local 'ceph_args=--fsid=973e3fc1-e63e-4bde-9db6-84c68fd5e93e --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2019-12-21T15:57:26.998 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2019-12-21T15:57:26.998 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-journal-size=100' 2019-12-21T15:57:26.998 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2019-12-21T15:57:26.998 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2019-12-21T15:57:26.998 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2019-12-21T15:57:26.998 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --chdir=' 2019-12-21T15:57:26.998 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+= 2019-12-21T15:57:26.999 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2019-12-21T15:57:26.999 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: get_asok_path 2019-12-21T15:57:26.999 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2019-12-21T15:57:26.999 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2019-12-21T15:57:26.999 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2019-12-21T15:57:26.999 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:57:26.999 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.119575 2019-12-21T15:57:27.000 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.119575/$cluster-$name.asok' 2019-12-21T15:57:27.000 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.119575/$cluster-$name.asok' 2019-12-21T15:57:27.000 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --debug-osd=20' 2019-12-21T15:57:27.000 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2019-12-21T15:57:27.000 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2019-12-21T15:57:27.000 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2019-12-21T15:57:27.000 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2019-12-21T15:57:27.001 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2019-12-21T15:57:27.001 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' ' 2019-12-21T15:57:27.001 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+= 2019-12-21T15:57:27.001 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: mkdir -p td/osd-scrub-repair/0 2019-12-21T15:57:27.001 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: echo start osd.0 2019-12-21T15:57:27.001 INFO:tasks.workunit.client.0.smithi172.stderr:start osd.0 2019-12-21T15:57:27.001 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph-osd -i 0 --fsid=973e3fc1-e63e-4bde-9db6-84c68fd5e93e --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.119575/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2019-12-21T15:57:27.002 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: cat td/osd-scrub-repair/0/whoami 2019-12-21T15:57:27.002 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: '[' 0 = 0 ']' 2019-12-21T15:57:27.002 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph osd dump --format=json 2019-12-21T15:57:27.003 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: jq '.flags_set[]' 2019-12-21T15:57:27.003 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: grep -q '"noup"' 2019-12-21T15:57:27.016 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:57:27.015+0000 7f24cd4eeec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T15:57:27.018 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:57:27.016+0000 7f24cd4eeec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T15:57:27.020 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:57:27.018+0000 7f24cd4eeec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T15:57:27.228 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: wait_for_osd up 0 2019-12-21T15:57:27.229 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:914: wait_for_osd: local state=up 2019-12-21T15:57:27.229 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local id=0 2019-12-21T15:57:27.229 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: status=1 2019-12-21T15:57:27.229 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i=0 )) 2019-12-21T15:57:27.229 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i < 300 )) 2019-12-21T15:57:27.229 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: echo 0 2019-12-21T15:57:27.230 INFO:tasks.workunit.client.0.smithi172.stderr:0 2019-12-21T15:57:27.230 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: ceph osd dump 2019-12-21T15:57:27.230 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: grep 'osd.0 up' 2019-12-21T15:57:27.454 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: sleep 1 2019-12-21T15:57:27.535 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:57:27.533+0000 7f24cd4eeec0 -1 Falling back to public interface 2019-12-21T15:57:28.334 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:57:28.332+0000 7f24cd4eeec0 -1 osd.0 21 log_to_monitors {default=true} 2019-12-21T15:57:28.456 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i++ )) 2019-12-21T15:57:28.456 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i < 300 )) 2019-12-21T15:57:28.456 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: echo 1 2019-12-21T15:57:28.456 INFO:tasks.workunit.client.0.smithi172.stderr:1 2019-12-21T15:57:28.456 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: grep 'osd.0 up' 2019-12-21T15:57:28.457 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: ceph osd dump 2019-12-21T15:57:28.704 INFO:tasks.workunit.client.0.smithi172.stderr:osd.0 up in weight 1 up_from 24 up_thru 24 down_at 22 last_clean_interval [5,21) [v2:127.0.0.1:6802/121155,v1:127.0.0.1:6803/121155] [v2:127.0.0.1:6804/121155,v1:127.0.0.1:6805/121155] exists,up ee9f2032-bba1-4e90-b9dd-383b0e131ab8 2019-12-21T15:57:28.705 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=0 2019-12-21T15:57:28.705 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: break 2019-12-21T15:57:28.705 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: return 0 2019-12-21T15:57:28.705 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1239: objectstore_tool: wait_for_clean 2019-12-21T15:57:28.705 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1548: wait_for_clean: local cmd= 2019-12-21T15:57:28.705 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1549: wait_for_clean: local num_active_clean=-1 2019-12-21T15:57:28.706 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1550: wait_for_clean: local cur_active_clean 2019-12-21T15:57:28.706 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1551: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2019-12-21T15:57:28.706 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1551: wait_for_clean: get_timeout_delays 90 .1 2019-12-21T15:57:28.706 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1494: get_timeout_delays: shopt -q -o xtrace 2019-12-21T15:57:28.706 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1494: get_timeout_delays: echo true 2019-12-21T15:57:28.706 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1494: get_timeout_delays: local trace=true 2019-12-21T15:57:28.706 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1495: get_timeout_delays: true 2019-12-21T15:57:28.707 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1495: get_timeout_delays: shopt -u -o xtrace 2019-12-21T15:57:28.795 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1551: wait_for_clean: local -a delays 2019-12-21T15:57:28.795 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1552: wait_for_clean: local -i loop=0 2019-12-21T15:57:28.796 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1554: wait_for_clean: flush_pg_stats 2019-12-21T15:57:28.796 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1977: flush_pg_stats: local timeout=300 2019-12-21T15:57:28.796 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1979: flush_pg_stats: ceph osd ls 2019-12-21T15:57:29.023 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1979: flush_pg_stats: ids='0 2019-12-21T15:57:29.023 INFO:tasks.workunit.client.0.smithi172.stderr:1' 2019-12-21T15:57:29.023 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1980: flush_pg_stats: seqs= 2019-12-21T15:57:29.024 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: flush_pg_stats: for osd in $ids 2019-12-21T15:57:29.024 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2019-12-21T15:57:29.134 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: seq=103079215106 2019-12-21T15:57:29.134 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: flush_pg_stats: seqs=' 0-103079215106' 2019-12-21T15:57:29.134 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: flush_pg_stats: for osd in $ids 2019-12-21T15:57:29.135 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2019-12-21T15:57:29.256 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: seq=42949672965 2019-12-21T15:57:29.256 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: flush_pg_stats: seqs=' 0-103079215106 1-42949672965' 2019-12-21T15:57:29.256 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: flush_pg_stats: for s in $seqs 2019-12-21T15:57:29.257 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: echo 0-103079215106 2019-12-21T15:57:29.257 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: cut -d - -f 1 2019-12-21T15:57:29.258 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: osd=0 2019-12-21T15:57:29.258 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: echo 0-103079215106 2019-12-21T15:57:29.258 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: cut -d - -f 2 2019-12-21T15:57:29.259 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: seq=103079215106 2019-12-21T15:57:29.259 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1989: flush_pg_stats: echo 'waiting osd.0 seq 103079215106' 2019-12-21T15:57:29.259 INFO:tasks.workunit.client.0.smithi172.stderr:waiting osd.0 seq 103079215106 2019-12-21T15:57:29.259 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 0 2019-12-21T15:57:29.486 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 103079215106 -lt 103079215106 2019-12-21T15:57:29.486 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: flush_pg_stats: for s in $seqs 2019-12-21T15:57:29.486 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: echo 1-42949672965 2019-12-21T15:57:29.487 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: cut -d - -f 1 2019-12-21T15:57:29.487 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: osd=1 2019-12-21T15:57:29.488 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: echo 1-42949672965 2019-12-21T15:57:29.488 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: cut -d - -f 2 2019-12-21T15:57:29.488 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: seq=42949672965 2019-12-21T15:57:29.489 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1989: flush_pg_stats: echo 'waiting osd.1 seq 42949672965' 2019-12-21T15:57:29.489 INFO:tasks.workunit.client.0.smithi172.stderr:waiting osd.1 seq 42949672965 2019-12-21T15:57:29.489 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 1 2019-12-21T15:57:29.718 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 42949672964 -lt 42949672965 2019-12-21T15:57:29.719 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1991: flush_pg_stats: sleep 1 2019-12-21T15:57:30.718 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1992: flush_pg_stats: '[' 300 -eq 0 ']' 2019-12-21T15:57:30.719 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 1 2019-12-21T15:57:30.950 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 42949672964 -lt 42949672965 2019-12-21T15:57:30.950 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1991: flush_pg_stats: sleep 1 2019-12-21T15:57:31.956 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1992: flush_pg_stats: '[' 299 -eq 0 ']' 2019-12-21T15:57:31.956 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 1 2019-12-21T15:57:32.186 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 42949672965 -lt 42949672965 2019-12-21T15:57:32.187 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1555: wait_for_clean: get_num_pgs 2019-12-21T15:57:32.187 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: ceph --format json status 2019-12-21T15:57:32.188 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: jq .pgmap.num_pgs 2019-12-21T15:57:32.455 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1555: wait_for_clean: test 4 == 0 2019-12-21T15:57:32.455 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1559: wait_for_clean: true 2019-12-21T15:57:32.456 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1563: wait_for_clean: get_num_active_clean 2019-12-21T15:57:32.456 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: get_num_active_clean: local expression 2019-12-21T15:57:32.456 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2019-12-21T15:57:32.456 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("stale") | not)' 2019-12-21T15:57:32.456 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: ceph --format json pg dump pgs 2019-12-21T15:57:32.456 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2019-12-21T15:57:32.679 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1563: wait_for_clean: cur_active_clean=4 2019-12-21T15:57:32.679 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1564: wait_for_clean: get_num_pgs 2019-12-21T15:57:32.679 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: ceph --format json status 2019-12-21T15:57:32.680 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: jq .pgmap.num_pgs 2019-12-21T15:57:32.948 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1564: wait_for_clean: test 4 = 4 2019-12-21T15:57:32.949 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1564: wait_for_clean: break 2019-12-21T15:57:32.949 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: return 0 2019-12-21T15:57:32.949 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:243: corrupt_and_repair_one: get_pg rbd SOMETHING 2019-12-21T15:57:32.949 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1029: get_pg: local poolname=rbd 2019-12-21T15:57:32.949 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1030: get_pg: local objectname=SOMETHING 2019-12-21T15:57:32.949 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1032: get_pg: ceph --format json osd map rbd SOMETHING 2019-12-21T15:57:32.950 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1032: get_pg: jq -r .pgid 2019-12-21T15:57:33.179 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:243: corrupt_and_repair_one: local pg=1.3 2019-12-21T15:57:33.179 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:244: corrupt_and_repair_one: repair 1.3 2019-12-21T15:57:33.179 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: repair: local pgid=1.3 2019-12-21T15:57:33.179 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1665: repair: get_last_scrub_stamp 1.3 2019-12-21T15:57:33.179 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.3 2019-12-21T15:57:33.180 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:57:33.180 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:57:33.180 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2019-12-21T15:57:33.401 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1665: repair: local last_scrub=2019-12-21T15:57:19.727838+0000 2019-12-21T15:57:33.402 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1666: repair: ceph pg repair 1.3 2019-12-21T15:57:33.616 INFO:tasks.workunit.client.0.smithi172.stderr:instructing pg 1.3 on osd.1 to repair 2019-12-21T15:57:33.625 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: repair: wait_for_scrub 1.3 2019-12-21T15:57:19.727838+0000 2019-12-21T15:57:33.625 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1789: wait_for_scrub: local pgid=1.3 2019-12-21T15:57:33.625 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1790: wait_for_scrub: local last_scrub=2019-12-21T15:57:19.727838+0000 2019-12-21T15:57:33.625 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1791: wait_for_scrub: local sname=last_scrub_stamp 2019-12-21T15:57:33.626 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i=0 )) 2019-12-21T15:57:33.626 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:57:33.626 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2019-12-21T15:57:33.626 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.3 2019-12-21T15:57:33.626 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:57:33.626 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:57:33.626 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2019-12-21T15:57:33.847 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:57:19.727838+0000 '>' 2019-12-21T15:57:19.727838+0000 2019-12-21T15:57:33.848 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:57:34.849 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:57:34.849 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:57:34.850 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2019-12-21T15:57:34.850 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.3 2019-12-21T15:57:34.850 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:57:34.850 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:57:34.850 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2019-12-21T15:57:35.079 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:57:19.727838+0000 '>' 2019-12-21T15:57:19.727838+0000 2019-12-21T15:57:35.080 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:57:36.080 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:57:36.081 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:57:36.082 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2019-12-21T15:57:36.082 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.3 2019-12-21T15:57:36.082 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:57:36.083 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:57:36.083 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2019-12-21T15:57:36.314 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:57:33.770438+0000 '>' 2019-12-21T15:57:19.727838+0000 2019-12-21T15:57:36.314 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1795: wait_for_scrub: return 0 2019-12-21T15:57:36.314 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:248: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2019-12-21T15:57:36.315 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: objectstore_tool: local dir=td/osd-scrub-repair 2019-12-21T15:57:36.315 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: objectstore_tool: shift 2019-12-21T15:57:36.315 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1235: objectstore_tool: local id=1 2019-12-21T15:57:36.315 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1236: objectstore_tool: shift 2019-12-21T15:57:36.315 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1238: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING list-attrs 2019-12-21T15:57:36.315 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1205: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2019-12-21T15:57:36.315 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nowait: shift 2019-12-21T15:57:36.316 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1207: _objectstore_tool_nowait: local id=1 2019-12-21T15:57:36.316 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1208: _objectstore_tool_nowait: shift 2019-12-21T15:57:36.316 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1210: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2019-12-21T15:57:36.316 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2019-12-21T15:57:36.316 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2019-12-21T15:57:36.316 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2019-12-21T15:57:36.316 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2019-12-21T15:57:36.317 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2019-12-21T15:57:36.419 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2019-12-21T15:57:36.420 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1212: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING list-attrs 2019-12-21T15:57:36.420 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1193: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2019-12-21T15:57:36.420 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1194: _objectstore_tool_nodown: shift 2019-12-21T15:57:36.420 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1195: _objectstore_tool_nodown: local id=1 2019-12-21T15:57:36.420 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: _objectstore_tool_nodown: shift 2019-12-21T15:57:36.420 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2019-12-21T15:57:36.421 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING list-attrs 2019-12-21T15:57:36.991 INFO:tasks.workunit.client.0.smithi172.stderr:Error getting attr on : 1.3_head,#-3:c0000000:::scrub_1.3:head#, (61) No data available 2019-12-21T15:57:36.991 INFO:tasks.workunit.client.0.smithi172.stdout:_ 2019-12-21T15:57:36.992 INFO:tasks.workunit.client.0.smithi172.stdout:snapset 2019-12-21T15:57:37.514 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1213: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2019-12-21T15:57:37.515 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:842: activate_osd: local dir=td/osd-scrub-repair 2019-12-21T15:57:37.515 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: shift 2019-12-21T15:57:37.515 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: local id=1 2019-12-21T15:57:37.515 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: shift 2019-12-21T15:57:37.515 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local osd_data=td/osd-scrub-repair/1 2019-12-21T15:57:37.515 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local 'ceph_args=--fsid=973e3fc1-e63e-4bde-9db6-84c68fd5e93e --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2019-12-21T15:57:37.516 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2019-12-21T15:57:37.516 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-journal-size=100' 2019-12-21T15:57:37.516 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2019-12-21T15:57:37.516 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2019-12-21T15:57:37.516 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2019-12-21T15:57:37.516 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --chdir=' 2019-12-21T15:57:37.517 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+= 2019-12-21T15:57:37.517 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2019-12-21T15:57:37.517 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: get_asok_path 2019-12-21T15:57:37.517 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2019-12-21T15:57:37.517 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2019-12-21T15:57:37.517 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2019-12-21T15:57:37.518 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:57:37.518 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.119575 2019-12-21T15:57:37.518 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.119575/$cluster-$name.asok' 2019-12-21T15:57:37.518 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.119575/$cluster-$name.asok' 2019-12-21T15:57:37.518 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --debug-osd=20' 2019-12-21T15:57:37.518 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2019-12-21T15:57:37.519 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2019-12-21T15:57:37.519 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2019-12-21T15:57:37.519 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2019-12-21T15:57:37.519 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2019-12-21T15:57:37.519 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' ' 2019-12-21T15:57:37.519 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+= 2019-12-21T15:57:37.519 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: mkdir -p td/osd-scrub-repair/1 2019-12-21T15:57:37.520 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: echo start osd.1 2019-12-21T15:57:37.520 INFO:tasks.workunit.client.0.smithi172.stderr:start osd.1 2019-12-21T15:57:37.520 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph-osd -i 1 --fsid=973e3fc1-e63e-4bde-9db6-84c68fd5e93e --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.119575/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2019-12-21T15:57:37.520 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: cat td/osd-scrub-repair/1/whoami 2019-12-21T15:57:37.521 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: '[' 1 = 1 ']' 2019-12-21T15:57:37.521 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph osd dump --format=json 2019-12-21T15:57:37.521 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: jq '.flags_set[]' 2019-12-21T15:57:37.521 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: grep -q '"noup"' 2019-12-21T15:57:37.536 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:57:37.534+0000 7effb3456ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T15:57:37.538 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:57:37.537+0000 7effb3456ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T15:57:37.540 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:57:37.538+0000 7effb3456ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T15:57:37.754 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: wait_for_osd up 1 2019-12-21T15:57:37.754 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:914: wait_for_osd: local state=up 2019-12-21T15:57:37.754 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local id=1 2019-12-21T15:57:37.754 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: status=1 2019-12-21T15:57:37.755 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i=0 )) 2019-12-21T15:57:37.755 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i < 300 )) 2019-12-21T15:57:37.755 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: echo 0 2019-12-21T15:57:37.755 INFO:tasks.workunit.client.0.smithi172.stderr:0 2019-12-21T15:57:37.755 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: ceph osd dump 2019-12-21T15:57:37.755 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: grep 'osd.1 up' 2019-12-21T15:57:37.980 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: sleep 1 2019-12-21T15:57:38.055 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:57:38.054+0000 7effb3456ec0 -1 Falling back to public interface 2019-12-21T15:57:38.778 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:57:38.776+0000 7effb3456ec0 -1 osd.1 26 log_to_monitors {default=true} 2019-12-21T15:57:38.981 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i++ )) 2019-12-21T15:57:38.981 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i < 300 )) 2019-12-21T15:57:38.981 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: echo 1 2019-12-21T15:57:38.981 INFO:tasks.workunit.client.0.smithi172.stderr:1 2019-12-21T15:57:38.982 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: ceph osd dump 2019-12-21T15:57:38.982 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: grep 'osd.1 up' 2019-12-21T15:57:39.287 INFO:tasks.workunit.client.0.smithi172.stderr:osd.1 up in weight 1 up_from 29 up_thru 29 down_at 27 last_clean_interval [10,26) [v2:127.0.0.1:6810/121948,v1:127.0.0.1:6811/121948] [v2:127.0.0.1:6812/121948,v1:127.0.0.1:6813/121948] exists,up e1be02d3-7106-4dc1-95bf-dd477adfeec5 2019-12-21T15:57:39.288 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=0 2019-12-21T15:57:39.288 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: break 2019-12-21T15:57:39.288 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: return 0 2019-12-21T15:57:39.288 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1239: objectstore_tool: wait_for_clean 2019-12-21T15:57:39.288 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1548: wait_for_clean: local cmd= 2019-12-21T15:57:39.289 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1549: wait_for_clean: local num_active_clean=-1 2019-12-21T15:57:39.289 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1550: wait_for_clean: local cur_active_clean 2019-12-21T15:57:39.289 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1551: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2019-12-21T15:57:39.289 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1551: wait_for_clean: get_timeout_delays 90 .1 2019-12-21T15:57:39.289 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1494: get_timeout_delays: shopt -q -o xtrace 2019-12-21T15:57:39.289 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1494: get_timeout_delays: echo true 2019-12-21T15:57:39.289 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1494: get_timeout_delays: local trace=true 2019-12-21T15:57:39.290 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1495: get_timeout_delays: true 2019-12-21T15:57:39.290 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1495: get_timeout_delays: shopt -u -o xtrace 2019-12-21T15:57:39.379 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1551: wait_for_clean: local -a delays 2019-12-21T15:57:39.379 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1552: wait_for_clean: local -i loop=0 2019-12-21T15:57:39.379 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1554: wait_for_clean: flush_pg_stats 2019-12-21T15:57:39.379 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1977: flush_pg_stats: local timeout=300 2019-12-21T15:57:39.379 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1979: flush_pg_stats: ceph osd ls 2019-12-21T15:57:39.650 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1979: flush_pg_stats: ids='0 2019-12-21T15:57:39.650 INFO:tasks.workunit.client.0.smithi172.stderr:1' 2019-12-21T15:57:39.650 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1980: flush_pg_stats: seqs= 2019-12-21T15:57:39.650 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: flush_pg_stats: for osd in $ids 2019-12-21T15:57:39.650 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2019-12-21T15:57:39.760 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: seq=103079215109 2019-12-21T15:57:39.761 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: flush_pg_stats: seqs=' 0-103079215109' 2019-12-21T15:57:39.761 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: flush_pg_stats: for osd in $ids 2019-12-21T15:57:39.761 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2019-12-21T15:57:39.870 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: seq=124554051586 2019-12-21T15:57:39.871 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: flush_pg_stats: seqs=' 0-103079215109 1-124554051586' 2019-12-21T15:57:39.871 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: flush_pg_stats: for s in $seqs 2019-12-21T15:57:39.871 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: echo 0-103079215109 2019-12-21T15:57:39.872 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: cut -d - -f 1 2019-12-21T15:57:39.872 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: osd=0 2019-12-21T15:57:39.872 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: echo 0-103079215109 2019-12-21T15:57:39.873 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: cut -d - -f 2 2019-12-21T15:57:39.873 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: seq=103079215109 2019-12-21T15:57:39.873 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1989: flush_pg_stats: echo 'waiting osd.0 seq 103079215109' 2019-12-21T15:57:39.874 INFO:tasks.workunit.client.0.smithi172.stderr:waiting osd.0 seq 103079215109 2019-12-21T15:57:39.874 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 0 2019-12-21T15:57:40.098 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 103079215108 -lt 103079215109 2019-12-21T15:57:40.098 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1991: flush_pg_stats: sleep 1 2019-12-21T15:57:41.099 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1992: flush_pg_stats: '[' 300 -eq 0 ']' 2019-12-21T15:57:41.100 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 0 2019-12-21T15:57:41.346 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 103079215109 -lt 103079215109 2019-12-21T15:57:41.346 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: flush_pg_stats: for s in $seqs 2019-12-21T15:57:41.347 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: echo 1-124554051586 2019-12-21T15:57:41.347 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: cut -d - -f 1 2019-12-21T15:57:41.347 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: osd=1 2019-12-21T15:57:41.348 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: echo 1-124554051586 2019-12-21T15:57:41.348 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: cut -d - -f 2 2019-12-21T15:57:41.349 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: seq=124554051586 2019-12-21T15:57:41.349 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1989: flush_pg_stats: echo 'waiting osd.1 seq 124554051586' 2019-12-21T15:57:41.349 INFO:tasks.workunit.client.0.smithi172.stderr:waiting osd.1 seq 124554051586 2019-12-21T15:57:41.349 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 1 2019-12-21T15:57:41.577 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 124554051586 -lt 124554051586 2019-12-21T15:57:41.578 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1555: wait_for_clean: get_num_pgs 2019-12-21T15:57:41.578 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: ceph --format json status 2019-12-21T15:57:41.578 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: jq .pgmap.num_pgs 2019-12-21T15:57:41.847 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1555: wait_for_clean: test 4 == 0 2019-12-21T15:57:41.847 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1559: wait_for_clean: true 2019-12-21T15:57:41.847 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1563: wait_for_clean: get_num_active_clean 2019-12-21T15:57:41.847 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: get_num_active_clean: local expression 2019-12-21T15:57:41.847 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2019-12-21T15:57:41.848 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("stale") | not)' 2019-12-21T15:57:41.848 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: ceph --format json pg dump pgs 2019-12-21T15:57:41.848 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2019-12-21T15:57:42.068 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1563: wait_for_clean: cur_active_clean=4 2019-12-21T15:57:42.069 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1564: wait_for_clean: get_num_pgs 2019-12-21T15:57:42.069 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: ceph --format json status 2019-12-21T15:57:42.069 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: jq .pgmap.num_pgs 2019-12-21T15:57:42.337 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1564: wait_for_clean: test 4 = 4 2019-12-21T15:57:42.337 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1564: wait_for_clean: break 2019-12-21T15:57:42.337 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: return 0 2019-12-21T15:57:42.337 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:249: corrupt_and_repair_one: rados --pool rbd get SOMETHING td/osd-scrub-repair/COPY 2019-12-21T15:57:52.726 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:250: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2019-12-21T15:57:52.727 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:132: TEST_allow_repair_during_recovery: teardown td/osd-scrub-repair 2019-12-21T15:57:52.727 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2019-12-21T15:57:52.727 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2019-12-21T15:57:52.728 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2019-12-21T15:57:52.728 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2019-12-21T15:57:52.728 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2019-12-21T15:57:52.729 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2019-12-21T15:57:52.729 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2019-12-21T15:57:52.729 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2019-12-21T15:57:52.836 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2019-12-21T15:57:52.836 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2019-12-21T15:57:52.837 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2019-12-21T15:57:52.837 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2019-12-21T15:57:52.838 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2019-12-21T15:57:52.838 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2019-12-21T15:57:52.839 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2019-12-21T15:57:52.841 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local 'pattern=|/usr/lib/systemd/systemd-coredump %P %u %g %s %t %c %h %e' 2019-12-21T15:57:52.841 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' '|' = '|' ']' 2019-12-21T15:57:52.841 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:177: teardown: pattern= 2019-12-21T15:57:52.842 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2019-12-21T15:57:52.842 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname '' 2019-12-21T15:57:52.842 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls . 2019-12-21T15:57:52.843 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2019-12-21T15:57:52.843 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2019-12-21T15:57:52.848 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2019-12-21T15:57:52.849 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:57:52.849 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.119575 2019-12-21T15:57:52.849 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.119575 2019-12-21T15:57:52.851 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2019-12-21T15:57:52.851 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2019-12-21T15:57:52.851 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:62: run: for func in $funcs 2019-12-21T15:57:52.851 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: TEST_auto_repair_bluestore_basic td/osd-scrub-repair 2019-12-21T15:57:52.851 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:333: TEST_auto_repair_bluestore_basic: local dir=td/osd-scrub-repair 2019-12-21T15:57:52.852 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:334: TEST_auto_repair_bluestore_basic: local poolname=testpool 2019-12-21T15:57:52.852 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:337: TEST_auto_repair_bluestore_basic: setup td/osd-scrub-repair 2019-12-21T15:57:52.852 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2019-12-21T15:57:52.852 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2019-12-21T15:57:52.852 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2019-12-21T15:57:52.852 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2019-12-21T15:57:52.853 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2019-12-21T15:57:52.853 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2019-12-21T15:57:52.853 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2019-12-21T15:57:52.853 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2019-12-21T15:57:52.853 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2019-12-21T15:57:52.853 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2019-12-21T15:57:52.854 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2019-12-21T15:57:52.854 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2019-12-21T15:57:52.854 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2019-12-21T15:57:52.854 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2019-12-21T15:57:52.855 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2019-12-21T15:57:52.855 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2019-12-21T15:57:52.855 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2019-12-21T15:57:52.857 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local 'pattern=|/usr/lib/systemd/systemd-coredump %P %u %g %s %t %c %h %e' 2019-12-21T15:57:52.858 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' '|' = '|' ']' 2019-12-21T15:57:52.858 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:177: teardown: pattern= 2019-12-21T15:57:52.858 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2019-12-21T15:57:52.858 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname '' 2019-12-21T15:57:52.858 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls . 2019-12-21T15:57:52.859 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2019-12-21T15:57:52.859 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2019-12-21T15:57:52.861 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2019-12-21T15:57:52.861 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:57:52.861 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.119575 2019-12-21T15:57:52.861 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.119575 2019-12-21T15:57:52.862 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2019-12-21T15:57:52.862 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2019-12-21T15:57:52.862 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2019-12-21T15:57:52.863 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2019-12-21T15:57:52.863 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:57:52.864 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.119575 2019-12-21T15:57:52.864 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.119575 2019-12-21T15:57:52.864 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2019-12-21T15:57:52.866 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2019-12-21T15:57:52.866 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2019-12-21T15:57:52.866 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2019-12-21T15:57:52.866 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: TEST_auto_repair_bluestore_basic: run_mon td/osd-scrub-repair a 2019-12-21T15:57:52.866 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2019-12-21T15:57:52.866 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2019-12-21T15:57:52.867 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2019-12-21T15:57:52.867 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2019-12-21T15:57:52.867 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2019-12-21T15:57:52.867 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2019-12-21T15:57:53.160 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2019-12-21T15:57:53.161 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2019-12-21T15:57:53.161 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2019-12-21T15:57:53.161 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2019-12-21T15:57:53.161 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:57:53.161 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.119575 2019-12-21T15:57:53.161 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.119575/$cluster-$name.asok' 2019-12-21T15:57:53.162 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.119575/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 2019-12-21T15:57:53.336 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: cat 2019-12-21T15:57:53.337 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: get_config mon a fsid 2019-12-21T15:57:53.337 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: get_config: local daemon=mon 2019-12-21T15:57:53.337 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: get_config: local id=a 2019-12-21T15:57:53.338 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local config=fsid 2019-12-21T15:57:53.339 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: jq -r .fsid 2019-12-21T15:57:53.339 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: get_asok_path mon.a 2019-12-21T15:57:53.339 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2019-12-21T15:57:53.339 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2019-12-21T15:57:53.339 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:57:53.340 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:57:53.340 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.119575 2019-12-21T15:57:53.340 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.119575/ceph-mon.a.asok 2019-12-21T15:57:53.340 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: CEPH_ARGS= 2019-12-21T15:57:53.340 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: ceph --format json daemon /tmp/ceph-asok.119575/ceph-mon.a.asok config get fsid 2019-12-21T15:57:53.433 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: get_config mon a mon_host 2019-12-21T15:57:53.433 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: get_config: local daemon=mon 2019-12-21T15:57:53.434 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: get_config: local id=a 2019-12-21T15:57:53.434 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local config=mon_host 2019-12-21T15:57:53.434 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: jq -r .mon_host 2019-12-21T15:57:53.434 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: get_asok_path mon.a 2019-12-21T15:57:53.434 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2019-12-21T15:57:53.434 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2019-12-21T15:57:53.434 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:57:53.435 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:57:53.435 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.119575 2019-12-21T15:57:53.435 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.119575/ceph-mon.a.asok 2019-12-21T15:57:53.435 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: CEPH_ARGS= 2019-12-21T15:57:53.435 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: ceph --format json daemon /tmp/ceph-asok.119575/ceph-mon.a.asok config get mon_host 2019-12-21T15:57:53.522 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:339: TEST_auto_repair_bluestore_basic: run_mgr td/osd-scrub-repair x 2019-12-21T15:57:53.523 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:551: run_mgr: local dir=td/osd-scrub-repair 2019-12-21T15:57:53.523 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: shift 2019-12-21T15:57:53.523 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local id=x 2019-12-21T15:57:53.523 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2019-12-21T15:57:53.523 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local data=td/osd-scrub-repair/x 2019-12-21T15:57:53.523 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:558: run_mgr: get_asok_path 2019-12-21T15:57:53.524 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2019-12-21T15:57:53.524 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2019-12-21T15:57:53.524 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2019-12-21T15:57:53.524 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:57:53.524 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.119575 2019-12-21T15:57:53.524 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.119575/$cluster-$name.asok' 2019-12-21T15:57:53.525 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:558: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2019-12-21T15:57:53.525 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:558: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.119575/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2019-12-21T15:57:54.145 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:342: TEST_auto_repair_bluestore_basic: local 'ceph_osd_args=--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2019-12-21T15:57:54.146 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:343: TEST_auto_repair_bluestore_basic: seq 0 2 2019-12-21T15:57:54.149 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:343: TEST_auto_repair_bluestore_basic: for id in $(seq 0 2) 2019-12-21T15:57:54.149 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:344: TEST_auto_repair_bluestore_basic: run_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2019-12-21T15:57:54.149 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local dir=td/osd-scrub-repair 2019-12-21T15:57:54.149 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: shift 2019-12-21T15:57:54.149 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local id=0 2019-12-21T15:57:54.150 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2019-12-21T15:57:54.150 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local osd_data=td/osd-scrub-repair/0 2019-12-21T15:57:54.150 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local 'ceph_args=--fsid=973e3fc1-e63e-4bde-9db6-84c68fd5e93e --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2019-12-21T15:57:54.150 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2019-12-21T15:57:54.150 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-journal-size=100' 2019-12-21T15:57:54.150 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2019-12-21T15:57:54.150 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2019-12-21T15:57:54.151 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2019-12-21T15:57:54.151 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --chdir=' 2019-12-21T15:57:54.151 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+= 2019-12-21T15:57:54.151 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2019-12-21T15:57:54.151 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: get_asok_path 2019-12-21T15:57:54.151 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2019-12-21T15:57:54.151 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2019-12-21T15:57:54.152 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2019-12-21T15:57:54.152 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:57:54.152 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.119575 2019-12-21T15:57:54.152 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.119575/$cluster-$name.asok' 2019-12-21T15:57:54.152 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.119575/$cluster-$name.asok' 2019-12-21T15:57:54.152 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --debug-osd=20' 2019-12-21T15:57:54.153 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-ms=1' 2019-12-21T15:57:54.153 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-monc=20' 2019-12-21T15:57:54.153 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2019-12-21T15:57:54.153 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2019-12-21T15:57:54.153 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2019-12-21T15:57:54.153 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2019-12-21T15:57:54.153 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2019-12-21T15:57:54.154 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' ' 2019-12-21T15:57:54.154 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2019-12-21T15:57:54.154 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: mkdir -p td/osd-scrub-repair/0 2019-12-21T15:57:54.155 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: uuidgen 2019-12-21T15:57:54.155 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: local uuid=1ae93a90-3644-4907-a6e4-08b0470a2899 2019-12-21T15:57:54.155 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: echo 'add osd0 1ae93a90-3644-4907-a6e4-08b0470a2899' 2019-12-21T15:57:54.155 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: ceph-authtool --gen-print-key 2019-12-21T15:57:54.155 INFO:tasks.workunit.client.0.smithi172.stdout:add osd0 1ae93a90-3644-4907-a6e4-08b0470a2899 2019-12-21T15:57:54.171 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: OSD_SECRET=AQACQf5dyvUUChAAkf0TY8EZpm8QuShpAB0y0A== 2019-12-21T15:57:54.171 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo '{"cephx_secret": "AQACQf5dyvUUChAAkf0TY8EZpm8QuShpAB0y0A=="}' 2019-12-21T15:57:54.171 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph osd new 1ae93a90-3644-4907-a6e4-08b0470a2899 -i td/osd-scrub-repair/0/new.json 2019-12-21T15:57:54.435 INFO:tasks.workunit.client.0.smithi172.stdout:0 2019-12-21T15:57:54.445 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: rm td/osd-scrub-repair/0/new.json 2019-12-21T15:57:54.446 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-osd -i 0 --fsid=973e3fc1-e63e-4bde-9db6-84c68fd5e93e --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.119575/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQACQf5dyvUUChAAkf0TY8EZpm8QuShpAB0y0A== --osd-uuid 1ae93a90-3644-4907-a6e4-08b0470a2899 2019-12-21T15:57:54.469 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:57:54.466+0000 7f0e9c92dec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T15:57:54.470 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:57:54.468+0000 7f0e9c92dec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T15:57:54.472 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:57:54.469+0000 7f0e9c92dec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T15:57:54.473 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:57:54.470+0000 7f0e9c92dec0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2019-12-21T15:57:54.473 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:57:54.470+0000 7f0e9c92dec0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2019-12-21T15:57:54.473 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:57:54.470+0000 7f0e9c92dec0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2019-12-21T15:57:54.527 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:57:54.524+0000 7f0e9c92dec0 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2019-12-21T15:57:57.505 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2019-12-21T15:57:57.505 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: cat 2019-12-21T15:57:57.506 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: echo adding osd0 key to auth repository 2019-12-21T15:57:57.506 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2019-12-21T15:57:57.506 INFO:tasks.workunit.client.0.smithi172.stdout:adding osd0 key to auth repository 2019-12-21T15:57:57.703 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo start osd.0 2019-12-21T15:57:57.703 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph-osd -i 0 --fsid=973e3fc1-e63e-4bde-9db6-84c68fd5e93e --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.119575/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2019-12-21T15:57:57.703 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph osd dump --format=json 2019-12-21T15:57:57.704 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: jq '.flags_set[]' 2019-12-21T15:57:57.704 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: grep -q '"noup"' 2019-12-21T15:57:57.704 INFO:tasks.workunit.client.0.smithi172.stdout:start osd.0 2019-12-21T15:57:57.727 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:57:57.722+0000 7ff94fbb4ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T15:57:57.744 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:57:57.741+0000 7ff94fbb4ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T15:57:57.745 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:57:57.743+0000 7ff94fbb4ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T15:57:57.934 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: wait_for_osd up 0 2019-12-21T15:57:57.934 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:914: wait_for_osd: local state=up 2019-12-21T15:57:57.934 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local id=0 2019-12-21T15:57:57.934 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: status=1 2019-12-21T15:57:57.934 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i=0 )) 2019-12-21T15:57:57.935 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i < 300 )) 2019-12-21T15:57:57.935 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: echo 0 2019-12-21T15:57:57.935 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: ceph osd dump 2019-12-21T15:57:57.935 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: grep 'osd.0 up' 2019-12-21T15:57:57.935 INFO:tasks.workunit.client.0.smithi172.stdout:0 2019-12-21T15:57:58.161 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: sleep 1 2019-12-21T15:57:58.261 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:57:58.258+0000 7ff94fbb4ec0 -1 Falling back to public interface 2019-12-21T15:57:58.983 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:57:58.981+0000 7ff94fbb4ec0 -1 osd.0 0 log_to_monitors {default=true} 2019-12-21T15:57:59.162 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i++ )) 2019-12-21T15:57:59.162 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i < 300 )) 2019-12-21T15:57:59.162 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: echo 1 2019-12-21T15:57:59.163 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: ceph osd dump 2019-12-21T15:57:59.163 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: grep 'osd.0 up' 2019-12-21T15:57:59.163 INFO:tasks.workunit.client.0.smithi172.stdout:1 2019-12-21T15:57:59.456 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: sleep 1 2019-12-21T15:58:00.457 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i++ )) 2019-12-21T15:58:00.458 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i < 300 )) 2019-12-21T15:58:00.458 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: echo 2 2019-12-21T15:58:00.458 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: ceph osd dump 2019-12-21T15:58:00.458 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: grep 'osd.0 up' 2019-12-21T15:58:00.458 INFO:tasks.workunit.client.0.smithi172.stdout:2 2019-12-21T15:58:00.688 INFO:tasks.workunit.client.0.smithi172.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/122889,v1:127.0.0.1:6803/122889] [v2:127.0.0.1:6804/122889,v1:127.0.0.1:6805/122889] exists,up 1ae93a90-3644-4907-a6e4-08b0470a2899 2019-12-21T15:58:00.688 INFO:tasks.workunit.client.0.smithi172.stdout:add osd1 084b38c8-6cf4-4894-8573-84b52fbf40c9 2019-12-21T15:58:00.689 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=0 2019-12-21T15:58:00.689 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: break 2019-12-21T15:58:00.689 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: return 0 2019-12-21T15:58:00.689 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:343: TEST_auto_repair_bluestore_basic: for id in $(seq 0 2) 2019-12-21T15:58:00.689 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:344: TEST_auto_repair_bluestore_basic: run_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2019-12-21T15:58:00.689 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local dir=td/osd-scrub-repair 2019-12-21T15:58:00.690 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: shift 2019-12-21T15:58:00.690 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local id=1 2019-12-21T15:58:00.690 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2019-12-21T15:58:00.690 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local osd_data=td/osd-scrub-repair/1 2019-12-21T15:58:00.690 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local 'ceph_args=--fsid=973e3fc1-e63e-4bde-9db6-84c68fd5e93e --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2019-12-21T15:58:00.690 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2019-12-21T15:58:00.690 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-journal-size=100' 2019-12-21T15:58:00.690 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2019-12-21T15:58:00.691 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2019-12-21T15:58:00.691 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2019-12-21T15:58:00.691 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --chdir=' 2019-12-21T15:58:00.691 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+= 2019-12-21T15:58:00.691 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2019-12-21T15:58:00.691 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: get_asok_path 2019-12-21T15:58:00.691 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2019-12-21T15:58:00.692 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2019-12-21T15:58:00.692 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2019-12-21T15:58:00.692 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:58:00.692 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.119575 2019-12-21T15:58:00.692 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.119575/$cluster-$name.asok' 2019-12-21T15:58:00.692 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.119575/$cluster-$name.asok' 2019-12-21T15:58:00.692 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --debug-osd=20' 2019-12-21T15:58:00.693 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-ms=1' 2019-12-21T15:58:00.693 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-monc=20' 2019-12-21T15:58:00.693 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2019-12-21T15:58:00.704 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2019-12-21T15:58:00.704 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2019-12-21T15:58:00.705 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2019-12-21T15:58:00.705 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2019-12-21T15:58:00.705 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' ' 2019-12-21T15:58:00.705 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2019-12-21T15:58:00.705 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: mkdir -p td/osd-scrub-repair/1 2019-12-21T15:58:00.705 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: uuidgen 2019-12-21T15:58:00.706 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: local uuid=084b38c8-6cf4-4894-8573-84b52fbf40c9 2019-12-21T15:58:00.706 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: echo 'add osd1 084b38c8-6cf4-4894-8573-84b52fbf40c9' 2019-12-21T15:58:00.706 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: ceph-authtool --gen-print-key 2019-12-21T15:58:00.707 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: OSD_SECRET=AQAIQf5dANeoKRAAv+AYEqODT9nUtcHkrchm2A== 2019-12-21T15:58:00.707 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo '{"cephx_secret": "AQAIQf5dANeoKRAAv+AYEqODT9nUtcHkrchm2A=="}' 2019-12-21T15:58:00.707 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph osd new 084b38c8-6cf4-4894-8573-84b52fbf40c9 -i td/osd-scrub-repair/1/new.json 2019-12-21T15:58:01.016 INFO:tasks.workunit.client.0.smithi172.stdout:1 2019-12-21T15:58:01.026 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: rm td/osd-scrub-repair/1/new.json 2019-12-21T15:58:01.027 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-osd -i 1 --fsid=973e3fc1-e63e-4bde-9db6-84c68fd5e93e --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.119575/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQAIQf5dANeoKRAAv+AYEqODT9nUtcHkrchm2A== --osd-uuid 084b38c8-6cf4-4894-8573-84b52fbf40c9 2019-12-21T15:58:01.050 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:58:01.047+0000 7f4e9abb8ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T15:58:01.051 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:58:01.049+0000 7f4e9abb8ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T15:58:01.052 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:58:01.050+0000 7f4e9abb8ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T15:58:01.053 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:58:01.050+0000 7f4e9abb8ec0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2019-12-21T15:58:01.053 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:58:01.050+0000 7f4e9abb8ec0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2019-12-21T15:58:01.053 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:58:01.050+0000 7f4e9abb8ec0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2019-12-21T15:58:01.116 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:58:01.114+0000 7f4e9abb8ec0 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2019-12-21T15:58:04.252 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2019-12-21T15:58:04.253 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: cat 2019-12-21T15:58:04.254 INFO:tasks.workunit.client.0.smithi172.stdout:adding osd1 key to auth repository 2019-12-21T15:58:04.254 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: echo adding osd1 key to auth repository 2019-12-21T15:58:04.254 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2019-12-21T15:58:04.537 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo start osd.1 2019-12-21T15:58:04.538 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph-osd -i 1 --fsid=973e3fc1-e63e-4bde-9db6-84c68fd5e93e --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.119575/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2019-12-21T15:58:04.538 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph osd dump --format=json 2019-12-21T15:58:04.538 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: jq '.flags_set[]' 2019-12-21T15:58:04.538 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: grep -q '"noup"' 2019-12-21T15:58:04.538 INFO:tasks.workunit.client.0.smithi172.stdout:start osd.1 2019-12-21T15:58:04.559 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:58:04.556+0000 7f1469a34ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T15:58:04.560 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:58:04.558+0000 7f1469a34ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T15:58:04.561 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:58:04.560+0000 7f1469a34ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T15:58:04.766 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: wait_for_osd up 1 2019-12-21T15:58:04.766 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:914: wait_for_osd: local state=up 2019-12-21T15:58:04.766 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local id=1 2019-12-21T15:58:04.766 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: status=1 2019-12-21T15:58:04.766 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i=0 )) 2019-12-21T15:58:04.766 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i < 300 )) 2019-12-21T15:58:04.767 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: echo 0 2019-12-21T15:58:04.767 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: ceph osd dump 2019-12-21T15:58:04.767 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: grep 'osd.1 up' 2019-12-21T15:58:04.767 INFO:tasks.workunit.client.0.smithi172.stdout:0 2019-12-21T15:58:04.993 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: sleep 1 2019-12-21T15:58:05.077 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:58:05.076+0000 7f1469a34ec0 -1 Falling back to public interface 2019-12-21T15:58:05.814 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:58:05.813+0000 7f1469a34ec0 -1 osd.1 0 log_to_monitors {default=true} 2019-12-21T15:58:05.994 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i++ )) 2019-12-21T15:58:05.994 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i < 300 )) 2019-12-21T15:58:05.994 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: echo 1 2019-12-21T15:58:05.995 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: ceph osd dump 2019-12-21T15:58:05.995 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: grep 'osd.1 up' 2019-12-21T15:58:05.995 INFO:tasks.workunit.client.0.smithi172.stdout:1 2019-12-21T15:58:06.237 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: sleep 1 2019-12-21T15:58:07.238 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i++ )) 2019-12-21T15:58:07.239 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i < 300 )) 2019-12-21T15:58:07.239 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: echo 2 2019-12-21T15:58:07.239 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: ceph osd dump 2019-12-21T15:58:07.239 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: grep 'osd.1 up' 2019-12-21T15:58:07.239 INFO:tasks.workunit.client.0.smithi172.stdout:2 2019-12-21T15:58:07.478 INFO:tasks.workunit.client.0.smithi172.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/123266,v1:127.0.0.1:6811/123266] [v2:127.0.0.1:6812/123266,v1:127.0.0.1:6813/123266] exists,up 084b38c8-6cf4-4894-8573-84b52fbf40c9 2019-12-21T15:58:07.478 INFO:tasks.workunit.client.0.smithi172.stdout:add osd2 f7bf09ae-8dcc-4261-94a5-92cf15967737 2019-12-21T15:58:07.479 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=0 2019-12-21T15:58:07.479 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: break 2019-12-21T15:58:07.479 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: return 0 2019-12-21T15:58:07.479 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:343: TEST_auto_repair_bluestore_basic: for id in $(seq 0 2) 2019-12-21T15:58:07.479 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:344: TEST_auto_repair_bluestore_basic: run_osd td/osd-scrub-repair 2 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2019-12-21T15:58:07.479 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local dir=td/osd-scrub-repair 2019-12-21T15:58:07.480 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: shift 2019-12-21T15:58:07.480 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local id=2 2019-12-21T15:58:07.480 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2019-12-21T15:58:07.480 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local osd_data=td/osd-scrub-repair/2 2019-12-21T15:58:07.480 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local 'ceph_args=--fsid=973e3fc1-e63e-4bde-9db6-84c68fd5e93e --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2019-12-21T15:58:07.480 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2019-12-21T15:58:07.480 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-journal-size=100' 2019-12-21T15:58:07.481 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2019-12-21T15:58:07.481 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2019-12-21T15:58:07.481 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2019-12-21T15:58:07.481 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --chdir=' 2019-12-21T15:58:07.481 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+= 2019-12-21T15:58:07.481 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2019-12-21T15:58:07.483 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: get_asok_path 2019-12-21T15:58:07.483 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2019-12-21T15:58:07.483 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2019-12-21T15:58:07.484 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2019-12-21T15:58:07.484 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:58:07.484 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.119575 2019-12-21T15:58:07.484 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.119575/$cluster-$name.asok' 2019-12-21T15:58:07.485 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.119575/$cluster-$name.asok' 2019-12-21T15:58:07.485 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --debug-osd=20' 2019-12-21T15:58:07.485 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-ms=1' 2019-12-21T15:58:07.485 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-monc=20' 2019-12-21T15:58:07.486 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2019-12-21T15:58:07.486 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2019-12-21T15:58:07.486 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2019-12-21T15:58:07.486 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2019-12-21T15:58:07.486 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2019-12-21T15:58:07.486 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' ' 2019-12-21T15:58:07.486 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2019-12-21T15:58:07.487 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: mkdir -p td/osd-scrub-repair/2 2019-12-21T15:58:07.487 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: uuidgen 2019-12-21T15:58:07.487 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: local uuid=f7bf09ae-8dcc-4261-94a5-92cf15967737 2019-12-21T15:58:07.487 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: echo 'add osd2 f7bf09ae-8dcc-4261-94a5-92cf15967737' 2019-12-21T15:58:07.487 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: ceph-authtool --gen-print-key 2019-12-21T15:58:07.490 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: OSD_SECRET=AQAPQf5ddV0iHRAAqxJV0lat1wNQbkA3QXM70g== 2019-12-21T15:58:07.490 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo '{"cephx_secret": "AQAPQf5ddV0iHRAAqxJV0lat1wNQbkA3QXM70g=="}' 2019-12-21T15:58:07.491 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph osd new f7bf09ae-8dcc-4261-94a5-92cf15967737 -i td/osd-scrub-repair/2/new.json 2019-12-21T15:58:07.780 INFO:tasks.workunit.client.0.smithi172.stdout:2 2019-12-21T15:58:07.790 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: rm td/osd-scrub-repair/2/new.json 2019-12-21T15:58:07.791 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-osd -i 2 --fsid=973e3fc1-e63e-4bde-9db6-84c68fd5e93e --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.119575/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQAPQf5ddV0iHRAAqxJV0lat1wNQbkA3QXM70g== --osd-uuid f7bf09ae-8dcc-4261-94a5-92cf15967737 2019-12-21T15:58:07.814 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:58:07.812+0000 7f7679b6aec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T15:58:07.816 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:58:07.814+0000 7f7679b6aec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T15:58:07.818 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:58:07.815+0000 7f7679b6aec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T15:58:07.818 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:58:07.816+0000 7f7679b6aec0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2019-12-21T15:58:07.818 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:58:07.816+0000 7f7679b6aec0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2019-12-21T15:58:07.818 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:58:07.816+0000 7f7679b6aec0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2019-12-21T15:58:07.880 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:58:07.879+0000 7f7679b6aec0 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2019-12-21T15:58:10.823 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2019-12-21T15:58:10.823 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: cat 2019-12-21T15:58:10.824 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: echo adding osd2 key to auth repository 2019-12-21T15:58:10.824 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2019-12-21T15:58:10.824 INFO:tasks.workunit.client.0.smithi172.stdout:adding osd2 key to auth repository 2019-12-21T15:58:11.111 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo start osd.2 2019-12-21T15:58:11.111 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph-osd -i 2 --fsid=973e3fc1-e63e-4bde-9db6-84c68fd5e93e --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.119575/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2019-12-21T15:58:11.111 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: jq '.flags_set[]' 2019-12-21T15:58:11.111 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph osd dump --format=json 2019-12-21T15:58:11.111 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: grep -q '"noup"' 2019-12-21T15:58:11.111 INFO:tasks.workunit.client.0.smithi172.stdout:start osd.2 2019-12-21T15:58:11.132 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:58:11.130+0000 7f06cb17eec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T15:58:11.133 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:58:11.132+0000 7f06cb17eec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T15:58:11.135 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:58:11.134+0000 7f06cb17eec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T15:58:11.339 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: wait_for_osd up 2 2019-12-21T15:58:11.339 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:914: wait_for_osd: local state=up 2019-12-21T15:58:11.339 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local id=2 2019-12-21T15:58:11.340 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: status=1 2019-12-21T15:58:11.340 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i=0 )) 2019-12-21T15:58:11.340 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i < 300 )) 2019-12-21T15:58:11.340 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: echo 0 2019-12-21T15:58:11.340 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: ceph osd dump 2019-12-21T15:58:11.340 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: grep 'osd.2 up' 2019-12-21T15:58:11.341 INFO:tasks.workunit.client.0.smithi172.stdout:0 2019-12-21T15:58:11.574 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: sleep 1 2019-12-21T15:58:11.651 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:58:11.649+0000 7f06cb17eec0 -1 Falling back to public interface 2019-12-21T15:58:12.379 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:58:12.378+0000 7f06cb17eec0 -1 osd.2 0 log_to_monitors {default=true} 2019-12-21T15:58:12.575 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i++ )) 2019-12-21T15:58:12.576 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i < 300 )) 2019-12-21T15:58:12.576 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: echo 1 2019-12-21T15:58:12.576 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: ceph osd dump 2019-12-21T15:58:12.576 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: grep 'osd.2 up' 2019-12-21T15:58:12.576 INFO:tasks.workunit.client.0.smithi172.stdout:1 2019-12-21T15:58:12.834 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: sleep 1 2019-12-21T15:58:13.836 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i++ )) 2019-12-21T15:58:13.836 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i < 300 )) 2019-12-21T15:58:13.836 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: echo 2 2019-12-21T15:58:13.836 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: ceph osd dump 2019-12-21T15:58:13.836 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: grep 'osd.2 up' 2019-12-21T15:58:13.837 INFO:tasks.workunit.client.0.smithi172.stdout:2 2019-12-21T15:58:14.064 INFO:tasks.workunit.client.0.smithi172.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/123643,v1:127.0.0.1:6819/123643] [v2:127.0.0.1:6820/123643,v1:127.0.0.1:6821/123643] exists,up f7bf09ae-8dcc-4261-94a5-92cf15967737 2019-12-21T15:58:14.065 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=0 2019-12-21T15:58:14.065 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: break 2019-12-21T15:58:14.065 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: return 0 2019-12-21T15:58:14.065 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:347: TEST_auto_repair_bluestore_basic: create_pool testpool 1 1 2019-12-21T15:58:14.065 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:539: create_pool: ceph osd pool create testpool 1 1 2019-12-21T15:58:14.448 INFO:tasks.workunit.client.0.smithi172.stderr:pool 'testpool' created 2019-12-21T15:58:14.455 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: sleep 1 2019-12-21T15:58:15.456 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:348: TEST_auto_repair_bluestore_basic: ceph osd pool set testpool size 2 2019-12-21T15:58:15.809 INFO:tasks.workunit.client.0.smithi172.stderr:set pool 1 size to 2 2019-12-21T15:58:15.817 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:349: TEST_auto_repair_bluestore_basic: wait_for_clean 2019-12-21T15:58:15.817 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1548: wait_for_clean: local cmd= 2019-12-21T15:58:15.818 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1549: wait_for_clean: local num_active_clean=-1 2019-12-21T15:58:15.818 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1550: wait_for_clean: local cur_active_clean 2019-12-21T15:58:15.818 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1551: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2019-12-21T15:58:15.818 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1551: wait_for_clean: get_timeout_delays 90 .1 2019-12-21T15:58:15.819 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1494: get_timeout_delays: shopt -q -o xtrace 2019-12-21T15:58:15.819 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1494: get_timeout_delays: echo true 2019-12-21T15:58:15.819 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1494: get_timeout_delays: local trace=true 2019-12-21T15:58:15.819 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1495: get_timeout_delays: true 2019-12-21T15:58:15.819 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1495: get_timeout_delays: shopt -u -o xtrace 2019-12-21T15:58:15.910 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1551: wait_for_clean: local -a delays 2019-12-21T15:58:15.910 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1552: wait_for_clean: local -i loop=0 2019-12-21T15:58:15.910 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1554: wait_for_clean: flush_pg_stats 2019-12-21T15:58:15.910 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1977: flush_pg_stats: local timeout=300 2019-12-21T15:58:15.910 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1979: flush_pg_stats: ceph osd ls 2019-12-21T15:58:16.139 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1979: flush_pg_stats: ids='0 2019-12-21T15:58:16.139 INFO:tasks.workunit.client.0.smithi172.stderr:1 2019-12-21T15:58:16.139 INFO:tasks.workunit.client.0.smithi172.stderr:2' 2019-12-21T15:58:16.139 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1980: flush_pg_stats: seqs= 2019-12-21T15:58:16.140 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: flush_pg_stats: for osd in $ids 2019-12-21T15:58:16.140 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2019-12-21T15:58:16.249 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: seq=21474836485 2019-12-21T15:58:16.249 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: flush_pg_stats: seqs=' 0-21474836485' 2019-12-21T15:58:16.249 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: flush_pg_stats: for osd in $ids 2019-12-21T15:58:16.249 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2019-12-21T15:58:16.360 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: seq=42949672963 2019-12-21T15:58:16.360 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: flush_pg_stats: seqs=' 0-21474836485 1-42949672963' 2019-12-21T15:58:16.360 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: flush_pg_stats: for osd in $ids 2019-12-21T15:58:16.361 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2019-12-21T15:58:16.471 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: seq=64424509442 2019-12-21T15:58:16.471 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: flush_pg_stats: seqs=' 0-21474836485 1-42949672963 2-64424509442' 2019-12-21T15:58:16.472 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: flush_pg_stats: for s in $seqs 2019-12-21T15:58:16.472 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: echo 0-21474836485 2019-12-21T15:58:16.472 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: cut -d - -f 1 2019-12-21T15:58:16.473 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: osd=0 2019-12-21T15:58:16.473 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: echo 0-21474836485 2019-12-21T15:58:16.473 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: cut -d - -f 2 2019-12-21T15:58:16.474 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: seq=21474836485 2019-12-21T15:58:16.474 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1989: flush_pg_stats: echo 'waiting osd.0 seq 21474836485' 2019-12-21T15:58:16.474 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 0 2019-12-21T15:58:16.475 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.0 seq 21474836485 2019-12-21T15:58:16.700 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 21474836484 -lt 21474836485 2019-12-21T15:58:16.700 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1991: flush_pg_stats: sleep 1 2019-12-21T15:58:17.701 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1992: flush_pg_stats: '[' 300 -eq 0 ']' 2019-12-21T15:58:17.701 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 0 2019-12-21T15:58:17.942 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 21474836485 -lt 21474836485 2019-12-21T15:58:17.942 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: flush_pg_stats: for s in $seqs 2019-12-21T15:58:17.942 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: echo 1-42949672963 2019-12-21T15:58:17.942 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: cut -d - -f 1 2019-12-21T15:58:17.943 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: osd=1 2019-12-21T15:58:17.943 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: echo 1-42949672963 2019-12-21T15:58:17.943 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: cut -d - -f 2 2019-12-21T15:58:17.943 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: seq=42949672963 2019-12-21T15:58:17.943 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1989: flush_pg_stats: echo 'waiting osd.1 seq 42949672963' 2019-12-21T15:58:17.943 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 1 2019-12-21T15:58:17.944 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.1 seq 42949672963 2019-12-21T15:58:18.168 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 42949672964 -lt 42949672963 2019-12-21T15:58:18.168 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: flush_pg_stats: for s in $seqs 2019-12-21T15:58:18.168 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: echo 2-64424509442 2019-12-21T15:58:18.168 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: cut -d - -f 1 2019-12-21T15:58:18.169 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: osd=2 2019-12-21T15:58:18.169 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: echo 2-64424509442 2019-12-21T15:58:18.169 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: cut -d - -f 2 2019-12-21T15:58:18.171 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: seq=64424509442 2019-12-21T15:58:18.171 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1989: flush_pg_stats: echo 'waiting osd.2 seq 64424509442' 2019-12-21T15:58:18.171 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 2 2019-12-21T15:58:18.171 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.2 seq 64424509442 2019-12-21T15:58:18.398 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 64424509442 -lt 64424509442 2019-12-21T15:58:18.398 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1555: wait_for_clean: get_num_pgs 2019-12-21T15:58:18.399 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: ceph --format json status 2019-12-21T15:58:18.399 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: jq .pgmap.num_pgs 2019-12-21T15:58:18.671 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1555: wait_for_clean: test 1 == 0 2019-12-21T15:58:18.671 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1559: wait_for_clean: true 2019-12-21T15:58:18.671 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1563: wait_for_clean: get_num_active_clean 2019-12-21T15:58:18.671 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: get_num_active_clean: local expression 2019-12-21T15:58:18.671 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2019-12-21T15:58:18.672 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("stale") | not)' 2019-12-21T15:58:18.672 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: ceph --format json pg dump pgs 2019-12-21T15:58:18.672 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2019-12-21T15:58:18.893 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1563: wait_for_clean: cur_active_clean=1 2019-12-21T15:58:18.893 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1564: wait_for_clean: get_num_pgs 2019-12-21T15:58:18.894 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: ceph --format json status 2019-12-21T15:58:18.894 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: jq .pgmap.num_pgs 2019-12-21T15:58:19.164 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1564: wait_for_clean: test 1 = 1 2019-12-21T15:58:19.164 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1564: wait_for_clean: break 2019-12-21T15:58:19.164 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: return 0 2019-12-21T15:58:19.164 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:352: TEST_auto_repair_bluestore_basic: local payload=ABCDEF 2019-12-21T15:58:19.165 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:353: TEST_auto_repair_bluestore_basic: echo ABCDEF 2019-12-21T15:58:19.165 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:354: TEST_auto_repair_bluestore_basic: rados --pool testpool put SOMETHING td/osd-scrub-repair/ORIGINAL 2019-12-21T15:58:19.206 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:358: TEST_auto_repair_bluestore_basic: get_not_primary testpool SOMETHING 2019-12-21T15:58:19.206 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1165: get_not_primary: local poolname=testpool 2019-12-21T15:58:19.207 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1166: get_not_primary: local objectname=SOMETHING 2019-12-21T15:58:19.207 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: get_not_primary: get_primary testpool SOMETHING 2019-12-21T15:58:19.207 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_primary: local poolname=testpool 2019-12-21T15:58:19.207 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1133: get_primary: local objectname=SOMETHING 2019-12-21T15:58:19.208 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1135: get_primary: ceph --format json osd map testpool SOMETHING 2019-12-21T15:58:19.208 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1136: get_primary: jq .acting_primary 2019-12-21T15:58:19.436 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: get_not_primary: local primary=1 2019-12-21T15:58:19.437 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1169: get_not_primary: ceph --format json osd map testpool SOMETHING 2019-12-21T15:58:19.437 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1170: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2019-12-21T15:58:19.670 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:358: TEST_auto_repair_bluestore_basic: objectstore_tool td/osd-scrub-repair 0 SOMETHING remove 2019-12-21T15:58:19.683 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: objectstore_tool: local dir=td/osd-scrub-repair 2019-12-21T15:58:19.683 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: objectstore_tool: shift 2019-12-21T15:58:19.683 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1235: objectstore_tool: local id=0 2019-12-21T15:58:19.683 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1236: objectstore_tool: shift 2019-12-21T15:58:19.684 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1238: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING remove 2019-12-21T15:58:19.684 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1205: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2019-12-21T15:58:19.684 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nowait: shift 2019-12-21T15:58:19.684 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1207: _objectstore_tool_nowait: local id=0 2019-12-21T15:58:19.685 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1208: _objectstore_tool_nowait: shift 2019-12-21T15:58:19.685 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1210: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2019-12-21T15:58:19.685 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2019-12-21T15:58:19.685 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2019-12-21T15:58:19.686 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2019-12-21T15:58:19.686 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2019-12-21T15:58:19.686 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2019-12-21T15:58:19.774 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2019-12-21T15:58:19.774 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1212: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING remove 2019-12-21T15:58:19.775 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1193: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2019-12-21T15:58:19.775 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1194: _objectstore_tool_nodown: shift 2019-12-21T15:58:19.775 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1195: _objectstore_tool_nodown: local id=0 2019-12-21T15:58:19.775 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: _objectstore_tool_nodown: shift 2019-12-21T15:58:19.775 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2019-12-21T15:58:19.775 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING remove 2019-12-21T15:58:20.441 INFO:tasks.workunit.client.0.smithi172.stdout:remove #1:eb822e21:::SOMETHING:head# 2019-12-21T15:58:20.960 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1213: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2019-12-21T15:58:20.960 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:842: activate_osd: local dir=td/osd-scrub-repair 2019-12-21T15:58:20.960 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: shift 2019-12-21T15:58:20.960 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: local id=0 2019-12-21T15:58:20.960 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: shift 2019-12-21T15:58:20.961 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local osd_data=td/osd-scrub-repair/0 2019-12-21T15:58:20.961 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local 'ceph_args=--fsid=973e3fc1-e63e-4bde-9db6-84c68fd5e93e --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2019-12-21T15:58:20.961 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2019-12-21T15:58:20.961 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-journal-size=100' 2019-12-21T15:58:20.961 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2019-12-21T15:58:20.962 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2019-12-21T15:58:20.962 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2019-12-21T15:58:20.962 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --chdir=' 2019-12-21T15:58:20.962 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+= 2019-12-21T15:58:20.962 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2019-12-21T15:58:20.963 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: get_asok_path 2019-12-21T15:58:20.963 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2019-12-21T15:58:20.963 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2019-12-21T15:58:20.963 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2019-12-21T15:58:20.963 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:58:20.964 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.119575 2019-12-21T15:58:20.964 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.119575/$cluster-$name.asok' 2019-12-21T15:58:20.964 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.119575/$cluster-$name.asok' 2019-12-21T15:58:20.964 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --debug-osd=20' 2019-12-21T15:58:20.965 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2019-12-21T15:58:20.965 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2019-12-21T15:58:20.965 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2019-12-21T15:58:20.965 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2019-12-21T15:58:20.965 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2019-12-21T15:58:20.966 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' ' 2019-12-21T15:58:20.966 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2019-12-21T15:58:20.966 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: mkdir -p td/osd-scrub-repair/0 2019-12-21T15:58:20.967 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: echo start osd.0 2019-12-21T15:58:20.968 INFO:tasks.workunit.client.0.smithi172.stderr:start osd.0 2019-12-21T15:58:20.968 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph-osd -i 0 --fsid=973e3fc1-e63e-4bde-9db6-84c68fd5e93e --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.119575/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2019-12-21T15:58:20.968 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: cat td/osd-scrub-repair/0/whoami 2019-12-21T15:58:20.968 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: '[' 0 = 0 ']' 2019-12-21T15:58:20.969 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph osd dump --format=json 2019-12-21T15:58:20.969 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: jq '.flags_set[]' 2019-12-21T15:58:20.969 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: grep -q '"noup"' 2019-12-21T15:58:20.984 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:58:20.982+0000 7f13b4cc1ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T15:58:20.986 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:58:20.984+0000 7f13b4cc1ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T15:58:20.989 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:58:20.987+0000 7f13b4cc1ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T15:58:21.196 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: wait_for_osd up 0 2019-12-21T15:58:21.196 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:914: wait_for_osd: local state=up 2019-12-21T15:58:21.197 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local id=0 2019-12-21T15:58:21.197 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: status=1 2019-12-21T15:58:21.197 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i=0 )) 2019-12-21T15:58:21.197 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i < 300 )) 2019-12-21T15:58:21.197 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: echo 0 2019-12-21T15:58:21.197 INFO:tasks.workunit.client.0.smithi172.stderr:0 2019-12-21T15:58:21.197 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: ceph osd dump 2019-12-21T15:58:21.198 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: grep 'osd.0 up' 2019-12-21T15:58:21.424 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: sleep 1 2019-12-21T15:58:21.503 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:58:21.502+0000 7f13b4cc1ec0 -1 Falling back to public interface 2019-12-21T15:58:22.210 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:58:22.208+0000 7f13b4cc1ec0 -1 osd.0 21 log_to_monitors {default=true} 2019-12-21T15:58:22.425 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i++ )) 2019-12-21T15:58:22.426 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i < 300 )) 2019-12-21T15:58:22.426 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: echo 1 2019-12-21T15:58:22.426 INFO:tasks.workunit.client.0.smithi172.stderr:1 2019-12-21T15:58:22.426 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: ceph osd dump 2019-12-21T15:58:22.427 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: grep 'osd.0 up' 2019-12-21T15:58:22.685 INFO:tasks.workunit.client.0.smithi172.stderr:osd.0 up in weight 1 up_from 24 up_thru 0 down_at 22 last_clean_interval [5,21) [v2:127.0.0.1:6802/124460,v1:127.0.0.1:6803/124460] [v2:127.0.0.1:6804/124460,v1:127.0.0.1:6805/124460] exists,up 1ae93a90-3644-4907-a6e4-08b0470a2899 2019-12-21T15:58:22.686 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=0 2019-12-21T15:58:22.686 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: break 2019-12-21T15:58:22.686 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: return 0 2019-12-21T15:58:22.686 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1239: objectstore_tool: wait_for_clean 2019-12-21T15:58:22.686 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1548: wait_for_clean: local cmd= 2019-12-21T15:58:22.686 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1549: wait_for_clean: local num_active_clean=-1 2019-12-21T15:58:22.686 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1550: wait_for_clean: local cur_active_clean 2019-12-21T15:58:22.687 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1551: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2019-12-21T15:58:22.687 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1551: wait_for_clean: get_timeout_delays 90 .1 2019-12-21T15:58:22.687 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1494: get_timeout_delays: shopt -q -o xtrace 2019-12-21T15:58:22.687 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1494: get_timeout_delays: echo true 2019-12-21T15:58:22.687 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1494: get_timeout_delays: local trace=true 2019-12-21T15:58:22.687 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1495: get_timeout_delays: true 2019-12-21T15:58:22.687 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1495: get_timeout_delays: shopt -u -o xtrace 2019-12-21T15:58:22.777 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1551: wait_for_clean: local -a delays 2019-12-21T15:58:22.778 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1552: wait_for_clean: local -i loop=0 2019-12-21T15:58:22.778 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1554: wait_for_clean: flush_pg_stats 2019-12-21T15:58:22.778 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1977: flush_pg_stats: local timeout=300 2019-12-21T15:58:22.778 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1979: flush_pg_stats: ceph osd ls 2019-12-21T15:58:23.004 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1979: flush_pg_stats: ids='0 2019-12-21T15:58:23.004 INFO:tasks.workunit.client.0.smithi172.stderr:1 2019-12-21T15:58:23.004 INFO:tasks.workunit.client.0.smithi172.stderr:2' 2019-12-21T15:58:23.004 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1980: flush_pg_stats: seqs= 2019-12-21T15:58:23.004 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: flush_pg_stats: for osd in $ids 2019-12-21T15:58:23.004 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2019-12-21T15:58:23.116 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: seq=103079215106 2019-12-21T15:58:23.116 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: flush_pg_stats: seqs=' 0-103079215106' 2019-12-21T15:58:23.116 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: flush_pg_stats: for osd in $ids 2019-12-21T15:58:23.117 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2019-12-21T15:58:23.228 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: seq=42949672966 2019-12-21T15:58:23.228 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: flush_pg_stats: seqs=' 0-103079215106 1-42949672966' 2019-12-21T15:58:23.228 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: flush_pg_stats: for osd in $ids 2019-12-21T15:58:23.228 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2019-12-21T15:58:23.338 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: seq=64424509444 2019-12-21T15:58:23.338 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: flush_pg_stats: seqs=' 0-103079215106 1-42949672966 2-64424509444' 2019-12-21T15:58:23.338 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: flush_pg_stats: for s in $seqs 2019-12-21T15:58:23.339 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: echo 0-103079215106 2019-12-21T15:58:23.339 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: cut -d - -f 1 2019-12-21T15:58:23.340 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: osd=0 2019-12-21T15:58:23.340 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: echo 0-103079215106 2019-12-21T15:58:23.340 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: cut -d - -f 2 2019-12-21T15:58:23.340 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: seq=103079215106 2019-12-21T15:58:23.341 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1989: flush_pg_stats: echo 'waiting osd.0 seq 103079215106' 2019-12-21T15:58:23.341 INFO:tasks.workunit.client.0.smithi172.stderr:waiting osd.0 seq 103079215106 2019-12-21T15:58:23.341 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 0 2019-12-21T15:58:23.576 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 21474836485 -lt 103079215106 2019-12-21T15:58:23.576 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1991: flush_pg_stats: sleep 1 2019-12-21T15:58:24.577 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1992: flush_pg_stats: '[' 300 -eq 0 ']' 2019-12-21T15:58:24.577 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 0 2019-12-21T15:58:24.813 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 103079215106 -lt 103079215106 2019-12-21T15:58:24.814 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: flush_pg_stats: for s in $seqs 2019-12-21T15:58:24.814 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: echo 1-42949672966 2019-12-21T15:58:24.814 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: cut -d - -f 1 2019-12-21T15:58:24.814 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: osd=1 2019-12-21T15:58:24.815 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: echo 1-42949672966 2019-12-21T15:58:24.815 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: cut -d - -f 2 2019-12-21T15:58:24.816 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: seq=42949672966 2019-12-21T15:58:24.816 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1989: flush_pg_stats: echo 'waiting osd.1 seq 42949672966' 2019-12-21T15:58:24.816 INFO:tasks.workunit.client.0.smithi172.stderr:waiting osd.1 seq 42949672966 2019-12-21T15:58:24.816 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 1 2019-12-21T15:58:25.043 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 42949672966 -lt 42949672966 2019-12-21T15:58:25.044 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: flush_pg_stats: for s in $seqs 2019-12-21T15:58:25.044 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: echo 2-64424509444 2019-12-21T15:58:25.044 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: cut -d - -f 1 2019-12-21T15:58:25.045 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: osd=2 2019-12-21T15:58:25.045 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: echo 2-64424509444 2019-12-21T15:58:25.045 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: cut -d - -f 2 2019-12-21T15:58:25.046 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: seq=64424509444 2019-12-21T15:58:25.046 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1989: flush_pg_stats: echo 'waiting osd.2 seq 64424509444' 2019-12-21T15:58:25.046 INFO:tasks.workunit.client.0.smithi172.stderr:waiting osd.2 seq 64424509444 2019-12-21T15:58:25.047 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 2 2019-12-21T15:58:25.279 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 64424509444 -lt 64424509444 2019-12-21T15:58:25.279 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1555: wait_for_clean: get_num_pgs 2019-12-21T15:58:25.280 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: ceph --format json status 2019-12-21T15:58:25.280 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: jq .pgmap.num_pgs 2019-12-21T15:58:25.542 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1555: wait_for_clean: test 1 == 0 2019-12-21T15:58:25.542 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1559: wait_for_clean: true 2019-12-21T15:58:25.542 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1563: wait_for_clean: get_num_active_clean 2019-12-21T15:58:25.543 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: get_num_active_clean: local expression 2019-12-21T15:58:25.543 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2019-12-21T15:58:25.543 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("stale") | not)' 2019-12-21T15:58:25.543 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: ceph --format json pg dump pgs 2019-12-21T15:58:25.543 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2019-12-21T15:58:25.767 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1563: wait_for_clean: cur_active_clean=1 2019-12-21T15:58:25.767 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1564: wait_for_clean: get_num_pgs 2019-12-21T15:58:25.767 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: ceph --format json status 2019-12-21T15:58:25.767 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: jq .pgmap.num_pgs 2019-12-21T15:58:26.037 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1564: wait_for_clean: test 1 = 1 2019-12-21T15:58:26.037 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1564: wait_for_clean: break 2019-12-21T15:58:26.037 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: return 0 2019-12-21T15:58:26.037 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:360: TEST_auto_repair_bluestore_basic: get_pg testpool SOMETHING 2019-12-21T15:58:26.037 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1029: get_pg: local poolname=testpool 2019-12-21T15:58:26.037 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1030: get_pg: local objectname=SOMETHING 2019-12-21T15:58:26.038 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1032: get_pg: ceph --format json osd map testpool SOMETHING 2019-12-21T15:58:26.038 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1032: get_pg: jq -r .pgid 2019-12-21T15:58:26.267 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:360: TEST_auto_repair_bluestore_basic: local pgid=1.0 2019-12-21T15:58:26.268 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:361: TEST_auto_repair_bluestore_basic: get_primary testpool SOMETHING 2019-12-21T15:58:26.268 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_primary: local poolname=testpool 2019-12-21T15:58:26.268 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1133: get_primary: local objectname=SOMETHING 2019-12-21T15:58:26.268 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1135: get_primary: ceph --format json osd map testpool SOMETHING 2019-12-21T15:58:26.268 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1136: get_primary: jq .acting_primary 2019-12-21T15:58:26.497 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:361: TEST_auto_repair_bluestore_basic: local primary=1 2019-12-21T15:58:26.497 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:362: TEST_auto_repair_bluestore_basic: get_last_scrub_stamp 1.0 2019-12-21T15:58:26.497 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.0 2019-12-21T15:58:26.498 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:58:26.498 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:58:26.498 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2019-12-21T15:58:26.719 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:362: TEST_auto_repair_bluestore_basic: local last_scrub_stamp=2019-12-21T15:58:14.369638+0000 2019-12-21T15:58:26.719 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:363: TEST_auto_repair_bluestore_basic: ceph tell 1.0 deep_scrub 2019-12-21T15:58:26.820 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:58:26.821 INFO:tasks.workunit.client.0.smithi172.stdout: "deep": true, 2019-12-21T15:58:26.821 INFO:tasks.workunit.client.0.smithi172.stdout: "stamp": "2019-12-14T15:56:46.819557+0000" 2019-12-21T15:58:26.821 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:58:26.833 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: TEST_auto_repair_bluestore_basic: ceph tell 1.0 scrub 2019-12-21T15:58:26.934 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:58:26.935 INFO:tasks.workunit.client.0.smithi172.stdout: "deep": false, 2019-12-21T15:58:26.935 INFO:tasks.workunit.client.0.smithi172.stdout: "stamp": "2019-12-14T15:56:46.933709+0000" 2019-12-21T15:58:26.935 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:58:26.944 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:367: TEST_auto_repair_bluestore_basic: wait_for_scrub 1.0 2019-12-21T15:58:14.369638+0000 2019-12-21T15:58:26.944 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1789: wait_for_scrub: local pgid=1.0 2019-12-21T15:58:26.944 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1790: wait_for_scrub: local last_scrub=2019-12-21T15:58:14.369638+0000 2019-12-21T15:58:26.944 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1791: wait_for_scrub: local sname=last_scrub_stamp 2019-12-21T15:58:26.944 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i=0 )) 2019-12-21T15:58:26.945 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:58:26.945 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2019-12-21T15:58:26.945 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.0 2019-12-21T15:58:26.945 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:58:26.945 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:58:26.945 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2019-12-21T15:58:27.166 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:58:14.369638+0000 '>' 2019-12-21T15:58:14.369638+0000 2019-12-21T15:58:27.166 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:58:28.168 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:58:28.168 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:58:28.168 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2019-12-21T15:58:28.169 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.0 2019-12-21T15:58:28.169 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:58:28.169 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:58:28.169 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2019-12-21T15:58:28.399 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-14T15:56:46.933709+0000 '>' 2019-12-21T15:58:14.369638+0000 2019-12-21T15:58:28.399 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:58:29.401 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:58:29.401 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:58:29.401 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2019-12-21T15:58:29.402 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.0 2019-12-21T15:58:29.402 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:58:29.402 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:58:29.402 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2019-12-21T15:58:29.635 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-14T15:56:46.933709+0000 '>' 2019-12-21T15:58:14.369638+0000 2019-12-21T15:58:29.635 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:58:30.636 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:58:30.636 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:58:30.636 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2019-12-21T15:58:30.637 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.0 2019-12-21T15:58:30.637 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:58:30.637 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:58:30.637 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2019-12-21T15:58:30.869 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-14T15:56:46.933709+0000 '>' 2019-12-21T15:58:14.369638+0000 2019-12-21T15:58:30.870 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:58:31.869 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:58:31.869 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:58:31.870 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2019-12-21T15:58:31.870 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.0 2019-12-21T15:58:31.870 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:58:31.871 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:58:31.871 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2019-12-21T15:58:32.103 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-14T15:56:46.933709+0000 '>' 2019-12-21T15:58:14.369638+0000 2019-12-21T15:58:32.104 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:58:33.104 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:58:33.104 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:58:33.105 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2019-12-21T15:58:33.106 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.0 2019-12-21T15:58:33.106 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:58:33.106 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:58:33.106 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2019-12-21T15:58:33.338 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-14T15:56:46.933709+0000 '>' 2019-12-21T15:58:14.369638+0000 2019-12-21T15:58:33.339 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:58:34.340 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:58:34.340 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:58:34.340 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2019-12-21T15:58:34.340 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.0 2019-12-21T15:58:34.340 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:58:34.341 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:58:34.341 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2019-12-21T15:58:34.565 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:58:28.892875+0000 '>' 2019-12-21T15:58:14.369638+0000 2019-12-21T15:58:34.565 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1795: wait_for_scrub: return 0 2019-12-21T15:58:34.566 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:368: TEST_auto_repair_bluestore_basic: wait_for_clean 2019-12-21T15:58:34.566 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1548: wait_for_clean: local cmd= 2019-12-21T15:58:34.566 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1549: wait_for_clean: local num_active_clean=-1 2019-12-21T15:58:34.566 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1550: wait_for_clean: local cur_active_clean 2019-12-21T15:58:34.566 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1551: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2019-12-21T15:58:34.566 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1551: wait_for_clean: get_timeout_delays 90 .1 2019-12-21T15:58:34.567 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1494: get_timeout_delays: shopt -q -o xtrace 2019-12-21T15:58:34.567 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1494: get_timeout_delays: echo true 2019-12-21T15:58:34.567 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1494: get_timeout_delays: local trace=true 2019-12-21T15:58:34.567 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1495: get_timeout_delays: true 2019-12-21T15:58:34.567 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1495: get_timeout_delays: shopt -u -o xtrace 2019-12-21T15:58:34.657 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1551: wait_for_clean: local -a delays 2019-12-21T15:58:34.658 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1552: wait_for_clean: local -i loop=0 2019-12-21T15:58:34.658 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1554: wait_for_clean: flush_pg_stats 2019-12-21T15:58:34.658 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1977: flush_pg_stats: local timeout=300 2019-12-21T15:58:34.658 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1979: flush_pg_stats: ceph osd ls 2019-12-21T15:58:34.884 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1979: flush_pg_stats: ids='0 2019-12-21T15:58:34.884 INFO:tasks.workunit.client.0.smithi172.stderr:1 2019-12-21T15:58:34.884 INFO:tasks.workunit.client.0.smithi172.stderr:2' 2019-12-21T15:58:34.884 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1980: flush_pg_stats: seqs= 2019-12-21T15:58:34.884 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: flush_pg_stats: for osd in $ids 2019-12-21T15:58:34.884 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2019-12-21T15:58:34.995 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: seq=103079215109 2019-12-21T15:58:34.995 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: flush_pg_stats: seqs=' 0-103079215109' 2019-12-21T15:58:34.995 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: flush_pg_stats: for osd in $ids 2019-12-21T15:58:34.995 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2019-12-21T15:58:35.106 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: seq=42949672969 2019-12-21T15:58:35.106 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: flush_pg_stats: seqs=' 0-103079215109 1-42949672969' 2019-12-21T15:58:35.106 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: flush_pg_stats: for osd in $ids 2019-12-21T15:58:35.106 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2019-12-21T15:58:35.215 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: seq=64424509448 2019-12-21T15:58:35.215 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: flush_pg_stats: seqs=' 0-103079215109 1-42949672969 2-64424509448' 2019-12-21T15:58:35.215 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: flush_pg_stats: for s in $seqs 2019-12-21T15:58:35.216 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: echo 0-103079215109 2019-12-21T15:58:35.216 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: cut -d - -f 1 2019-12-21T15:58:35.217 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: osd=0 2019-12-21T15:58:35.217 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: echo 0-103079215109 2019-12-21T15:58:35.217 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: cut -d - -f 2 2019-12-21T15:58:35.218 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: seq=103079215109 2019-12-21T15:58:35.218 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1989: flush_pg_stats: echo 'waiting osd.0 seq 103079215109' 2019-12-21T15:58:35.218 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.0 seq 103079215109 2019-12-21T15:58:35.218 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 0 2019-12-21T15:58:35.445 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 103079215108 -lt 103079215109 2019-12-21T15:58:35.445 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1991: flush_pg_stats: sleep 1 2019-12-21T15:58:36.446 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1992: flush_pg_stats: '[' 300 -eq 0 ']' 2019-12-21T15:58:36.447 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 0 2019-12-21T15:58:36.683 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 103079215109 -lt 103079215109 2019-12-21T15:58:36.683 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: flush_pg_stats: for s in $seqs 2019-12-21T15:58:36.683 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: echo 1-42949672969 2019-12-21T15:58:36.684 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: cut -d - -f 1 2019-12-21T15:58:36.684 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: osd=1 2019-12-21T15:58:36.684 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: echo 1-42949672969 2019-12-21T15:58:36.685 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: cut -d - -f 2 2019-12-21T15:58:36.693 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: seq=42949672969 2019-12-21T15:58:36.693 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1989: flush_pg_stats: echo 'waiting osd.1 seq 42949672969' 2019-12-21T15:58:36.694 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 1 2019-12-21T15:58:36.694 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.1 seq 42949672969 2019-12-21T15:58:36.913 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 42949672969 -lt 42949672969 2019-12-21T15:58:36.913 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: flush_pg_stats: for s in $seqs 2019-12-21T15:58:36.913 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: echo 2-64424509448 2019-12-21T15:58:36.914 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: cut -d - -f 1 2019-12-21T15:58:36.915 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: osd=2 2019-12-21T15:58:36.916 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: echo 2-64424509448 2019-12-21T15:58:36.916 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: cut -d - -f 2 2019-12-21T15:58:36.916 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: seq=64424509448 2019-12-21T15:58:36.916 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1989: flush_pg_stats: echo 'waiting osd.2 seq 64424509448' 2019-12-21T15:58:36.917 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 2 2019-12-21T15:58:36.917 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.2 seq 64424509448 2019-12-21T15:58:37.144 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 64424509448 -lt 64424509448 2019-12-21T15:58:37.144 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1555: wait_for_clean: get_num_pgs 2019-12-21T15:58:37.145 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: ceph --format json status 2019-12-21T15:58:37.145 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: jq .pgmap.num_pgs 2019-12-21T15:58:37.419 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1555: wait_for_clean: test 1 == 0 2019-12-21T15:58:37.420 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1559: wait_for_clean: true 2019-12-21T15:58:37.420 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1563: wait_for_clean: get_num_active_clean 2019-12-21T15:58:37.420 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: get_num_active_clean: local expression 2019-12-21T15:58:37.420 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2019-12-21T15:58:37.421 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("stale") | not)' 2019-12-21T15:58:37.421 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: ceph --format json pg dump pgs 2019-12-21T15:58:37.421 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2019-12-21T15:58:37.642 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1563: wait_for_clean: cur_active_clean=1 2019-12-21T15:58:37.642 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1564: wait_for_clean: get_num_pgs 2019-12-21T15:58:37.643 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: ceph --format json status 2019-12-21T15:58:37.643 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: jq .pgmap.num_pgs 2019-12-21T15:58:37.912 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1564: wait_for_clean: test 1 = 1 2019-12-21T15:58:37.912 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1564: wait_for_clean: break 2019-12-21T15:58:37.912 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: return 0 2019-12-21T15:58:37.912 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:369: TEST_auto_repair_bluestore_basic: ceph pg dump pgs 2019-12-21T15:58:38.124 INFO:tasks.workunit.client.0.smithi172.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2019-12-21T15:58:38.124 INFO:tasks.workunit.client.0.smithi172.stdout:1.0 1 0 0 0 0 7 0 0 1 1 active+clean 2019-12-21T15:58:29.118088+0000 21'1 25:49 [1,0] 1 [1,0] 1 21'1 2019-12-21T15:58:28.892875+0000 21'1 2019-12-21T15:58:28.892875+0000 0 2019-12-21T15:58:38.124 INFO:tasks.workunit.client.0.smithi172.stdout: 2019-12-21T15:58:38.124 INFO:tasks.workunit.client.0.smithi172.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilisation. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2019-12-21T15:58:38.125 INFO:tasks.workunit.client.0.smithi172.stderr:dumped pgs 2019-12-21T15:58:38.134 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:372: TEST_auto_repair_bluestore_basic: get_not_primary testpool SOMETHING 2019-12-21T15:58:38.135 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1165: get_not_primary: local poolname=testpool 2019-12-21T15:58:38.135 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1166: get_not_primary: local objectname=SOMETHING 2019-12-21T15:58:38.135 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: get_not_primary: get_primary testpool SOMETHING 2019-12-21T15:58:38.135 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_primary: local poolname=testpool 2019-12-21T15:58:38.135 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1133: get_primary: local objectname=SOMETHING 2019-12-21T15:58:38.135 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1135: get_primary: ceph --format json osd map testpool SOMETHING 2019-12-21T15:58:38.136 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1136: get_primary: jq .acting_primary 2019-12-21T15:58:38.364 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: get_not_primary: local primary=1 2019-12-21T15:58:38.364 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1169: get_not_primary: ceph --format json osd map testpool SOMETHING 2019-12-21T15:58:38.364 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1170: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2019-12-21T15:58:38.595 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:372: TEST_auto_repair_bluestore_basic: objectstore_tool td/osd-scrub-repair 0 SOMETHING list-attrs 2019-12-21T15:58:38.595 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: objectstore_tool: local dir=td/osd-scrub-repair 2019-12-21T15:58:38.595 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: objectstore_tool: shift 2019-12-21T15:58:38.595 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1235: objectstore_tool: local id=0 2019-12-21T15:58:38.596 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1236: objectstore_tool: shift 2019-12-21T15:58:38.596 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1238: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING list-attrs 2019-12-21T15:58:38.596 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1205: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2019-12-21T15:58:38.596 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nowait: shift 2019-12-21T15:58:38.596 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1207: _objectstore_tool_nowait: local id=0 2019-12-21T15:58:38.596 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1208: _objectstore_tool_nowait: shift 2019-12-21T15:58:38.597 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1210: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2019-12-21T15:58:38.597 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2019-12-21T15:58:38.597 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2019-12-21T15:58:38.597 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2019-12-21T15:58:38.597 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2019-12-21T15:58:38.597 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2019-12-21T15:58:38.699 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2019-12-21T15:58:38.700 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1212: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING list-attrs 2019-12-21T15:58:38.700 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1193: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2019-12-21T15:58:38.700 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1194: _objectstore_tool_nodown: shift 2019-12-21T15:58:38.700 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1195: _objectstore_tool_nodown: local id=0 2019-12-21T15:58:38.700 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: _objectstore_tool_nodown: shift 2019-12-21T15:58:38.700 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2019-12-21T15:58:38.701 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING list-attrs 2019-12-21T15:58:39.266 INFO:tasks.workunit.client.0.smithi172.stdout:_ 2019-12-21T15:58:39.266 INFO:tasks.workunit.client.0.smithi172.stdout:snapset 2019-12-21T15:58:39.785 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1213: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2019-12-21T15:58:39.785 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:842: activate_osd: local dir=td/osd-scrub-repair 2019-12-21T15:58:39.786 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: shift 2019-12-21T15:58:39.786 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: local id=0 2019-12-21T15:58:39.786 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: shift 2019-12-21T15:58:39.786 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local osd_data=td/osd-scrub-repair/0 2019-12-21T15:58:39.786 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local 'ceph_args=--fsid=973e3fc1-e63e-4bde-9db6-84c68fd5e93e --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2019-12-21T15:58:39.786 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2019-12-21T15:58:39.787 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-journal-size=100' 2019-12-21T15:58:39.787 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2019-12-21T15:58:39.787 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2019-12-21T15:58:39.787 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2019-12-21T15:58:39.787 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --chdir=' 2019-12-21T15:58:39.788 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+= 2019-12-21T15:58:39.788 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2019-12-21T15:58:39.788 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: get_asok_path 2019-12-21T15:58:39.789 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2019-12-21T15:58:39.789 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2019-12-21T15:58:39.789 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2019-12-21T15:58:39.789 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:58:39.790 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.119575 2019-12-21T15:58:39.790 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.119575/$cluster-$name.asok' 2019-12-21T15:58:39.790 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.119575/$cluster-$name.asok' 2019-12-21T15:58:39.790 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --debug-osd=20' 2019-12-21T15:58:39.790 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2019-12-21T15:58:39.791 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2019-12-21T15:58:39.791 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2019-12-21T15:58:39.791 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2019-12-21T15:58:39.791 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2019-12-21T15:58:39.791 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' ' 2019-12-21T15:58:39.791 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2019-12-21T15:58:39.792 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: mkdir -p td/osd-scrub-repair/0 2019-12-21T15:58:39.792 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: echo start osd.0 2019-12-21T15:58:39.792 INFO:tasks.workunit.client.0.smithi172.stderr:start osd.0 2019-12-21T15:58:39.792 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph-osd -i 0 --fsid=973e3fc1-e63e-4bde-9db6-84c68fd5e93e --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.119575/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2019-12-21T15:58:39.792 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: cat td/osd-scrub-repair/0/whoami 2019-12-21T15:58:39.793 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: '[' 0 = 0 ']' 2019-12-21T15:58:39.793 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph osd dump --format=json 2019-12-21T15:58:39.793 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: jq '.flags_set[]' 2019-12-21T15:58:39.795 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: grep -q '"noup"' 2019-12-21T15:58:39.807 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:58:39.804+0000 7f83ddc52ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T15:58:39.809 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:58:39.807+0000 7f83ddc52ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T15:58:39.812 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:58:39.809+0000 7f83ddc52ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T15:58:40.020 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: wait_for_osd up 0 2019-12-21T15:58:40.021 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:914: wait_for_osd: local state=up 2019-12-21T15:58:40.021 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local id=0 2019-12-21T15:58:40.021 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: status=1 2019-12-21T15:58:40.021 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i=0 )) 2019-12-21T15:58:40.021 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i < 300 )) 2019-12-21T15:58:40.021 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: echo 0 2019-12-21T15:58:40.022 INFO:tasks.workunit.client.0.smithi172.stderr:0 2019-12-21T15:58:40.023 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: grep 'osd.0 up' 2019-12-21T15:58:40.023 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: ceph osd dump 2019-12-21T15:58:40.246 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: sleep 1 2019-12-21T15:58:40.327 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:58:40.324+0000 7f83ddc52ec0 -1 Falling back to public interface 2019-12-21T15:58:41.060 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:58:41.057+0000 7f83ddc52ec0 -1 osd.0 25 log_to_monitors {default=true} 2019-12-21T15:58:41.248 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i++ )) 2019-12-21T15:58:41.248 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i < 300 )) 2019-12-21T15:58:41.248 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: echo 1 2019-12-21T15:58:41.248 INFO:tasks.workunit.client.0.smithi172.stderr:1 2019-12-21T15:58:41.249 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: ceph osd dump 2019-12-21T15:58:41.249 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: grep 'osd.0 up' 2019-12-21T15:58:41.483 INFO:tasks.workunit.client.0.smithi172.stderr:osd.0 up in weight 1 up_from 28 up_thru 0 down_at 26 last_clean_interval [24,25) [v2:127.0.0.1:6802/125930,v1:127.0.0.1:6803/125930] [v2:127.0.0.1:6804/125930,v1:127.0.0.1:6805/125930] exists,up 1ae93a90-3644-4907-a6e4-08b0470a2899 2019-12-21T15:58:41.483 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=0 2019-12-21T15:58:41.483 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: break 2019-12-21T15:58:41.484 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: return 0 2019-12-21T15:58:41.484 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1239: objectstore_tool: wait_for_clean 2019-12-21T15:58:41.484 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1548: wait_for_clean: local cmd= 2019-12-21T15:58:41.484 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1549: wait_for_clean: local num_active_clean=-1 2019-12-21T15:58:41.484 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1550: wait_for_clean: local cur_active_clean 2019-12-21T15:58:41.484 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1551: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2019-12-21T15:58:41.485 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1551: wait_for_clean: get_timeout_delays 90 .1 2019-12-21T15:58:41.485 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1494: get_timeout_delays: shopt -q -o xtrace 2019-12-21T15:58:41.485 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1494: get_timeout_delays: echo true 2019-12-21T15:58:41.485 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1494: get_timeout_delays: local trace=true 2019-12-21T15:58:41.485 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1495: get_timeout_delays: true 2019-12-21T15:58:41.485 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1495: get_timeout_delays: shopt -u -o xtrace 2019-12-21T15:58:41.575 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1551: wait_for_clean: local -a delays 2019-12-21T15:58:41.575 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1552: wait_for_clean: local -i loop=0 2019-12-21T15:58:41.575 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1554: wait_for_clean: flush_pg_stats 2019-12-21T15:58:41.576 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1977: flush_pg_stats: local timeout=300 2019-12-21T15:58:41.576 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1979: flush_pg_stats: ceph osd ls 2019-12-21T15:58:41.802 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1979: flush_pg_stats: ids='0 2019-12-21T15:58:41.802 INFO:tasks.workunit.client.0.smithi172.stderr:1 2019-12-21T15:58:41.802 INFO:tasks.workunit.client.0.smithi172.stderr:2' 2019-12-21T15:58:41.803 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1980: flush_pg_stats: seqs= 2019-12-21T15:58:41.803 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: flush_pg_stats: for osd in $ids 2019-12-21T15:58:41.803 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2019-12-21T15:58:41.914 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: seq=120259084290 2019-12-21T15:58:41.915 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: flush_pg_stats: seqs=' 0-120259084290' 2019-12-21T15:58:41.915 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: flush_pg_stats: for osd in $ids 2019-12-21T15:58:41.915 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2019-12-21T15:58:42.026 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: seq=42949672971 2019-12-21T15:58:42.026 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: flush_pg_stats: seqs=' 0-120259084290 1-42949672971' 2019-12-21T15:58:42.026 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: flush_pg_stats: for osd in $ids 2019-12-21T15:58:42.026 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2019-12-21T15:58:42.136 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: seq=64424509450 2019-12-21T15:58:42.137 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: flush_pg_stats: seqs=' 0-120259084290 1-42949672971 2-64424509450' 2019-12-21T15:58:42.137 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: flush_pg_stats: for s in $seqs 2019-12-21T15:58:42.137 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: echo 0-120259084290 2019-12-21T15:58:42.138 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: cut -d - -f 1 2019-12-21T15:58:42.138 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: osd=0 2019-12-21T15:58:42.138 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: echo 0-120259084290 2019-12-21T15:58:42.139 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: cut -d - -f 2 2019-12-21T15:58:42.141 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: seq=120259084290 2019-12-21T15:58:42.141 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1989: flush_pg_stats: echo 'waiting osd.0 seq 120259084290' 2019-12-21T15:58:42.141 INFO:tasks.workunit.client.0.smithi172.stderr:waiting osd.0 seq 120259084290 2019-12-21T15:58:42.141 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 0 2019-12-21T15:58:42.370 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 1 -lt 120259084290 2019-12-21T15:58:42.370 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1991: flush_pg_stats: sleep 1 2019-12-21T15:58:43.371 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1992: flush_pg_stats: '[' 300 -eq 0 ']' 2019-12-21T15:58:43.371 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 0 2019-12-21T15:58:43.638 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 1 -lt 120259084290 2019-12-21T15:58:43.638 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1991: flush_pg_stats: sleep 1 2019-12-21T15:58:44.639 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1992: flush_pg_stats: '[' 299 -eq 0 ']' 2019-12-21T15:58:44.639 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 0 2019-12-21T15:58:44.871 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 120259084290 -lt 120259084290 2019-12-21T15:58:44.872 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: flush_pg_stats: for s in $seqs 2019-12-21T15:58:44.872 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: echo 1-42949672971 2019-12-21T15:58:44.872 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: cut -d - -f 1 2019-12-21T15:58:44.873 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: osd=1 2019-12-21T15:58:44.873 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: echo 1-42949672971 2019-12-21T15:58:44.873 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: cut -d - -f 2 2019-12-21T15:58:44.874 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: seq=42949672971 2019-12-21T15:58:44.874 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1989: flush_pg_stats: echo 'waiting osd.1 seq 42949672971' 2019-12-21T15:58:44.875 INFO:tasks.workunit.client.0.smithi172.stderr:waiting osd.1 seq 42949672971 2019-12-21T15:58:44.875 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 1 2019-12-21T15:58:45.106 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 42949672972 -lt 42949672971 2019-12-21T15:58:45.107 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: flush_pg_stats: for s in $seqs 2019-12-21T15:58:45.107 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: echo 2-64424509450 2019-12-21T15:58:45.107 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: cut -d - -f 1 2019-12-21T15:58:45.108 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: osd=2 2019-12-21T15:58:45.108 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: echo 2-64424509450 2019-12-21T15:58:45.108 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: cut -d - -f 2 2019-12-21T15:58:45.109 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: seq=64424509450 2019-12-21T15:58:45.109 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1989: flush_pg_stats: echo 'waiting osd.2 seq 64424509450' 2019-12-21T15:58:45.109 INFO:tasks.workunit.client.0.smithi172.stderr:waiting osd.2 seq 64424509450 2019-12-21T15:58:45.110 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 2 2019-12-21T15:58:45.335 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 64424509450 -lt 64424509450 2019-12-21T15:58:45.336 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1555: wait_for_clean: get_num_pgs 2019-12-21T15:58:45.336 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: ceph --format json status 2019-12-21T15:58:45.336 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: jq .pgmap.num_pgs 2019-12-21T15:58:45.631 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1555: wait_for_clean: test 1 == 0 2019-12-21T15:58:45.631 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1559: wait_for_clean: true 2019-12-21T15:58:45.632 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1563: wait_for_clean: get_num_active_clean 2019-12-21T15:58:45.632 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: get_num_active_clean: local expression 2019-12-21T15:58:45.632 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2019-12-21T15:58:45.632 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("stale") | not)' 2019-12-21T15:58:45.632 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: ceph --format json pg dump pgs 2019-12-21T15:58:45.632 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2019-12-21T15:58:45.854 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1563: wait_for_clean: cur_active_clean=0 2019-12-21T15:58:45.855 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1564: wait_for_clean: get_num_pgs 2019-12-21T15:58:45.855 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: ceph --format json status 2019-12-21T15:58:45.855 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: jq .pgmap.num_pgs 2019-12-21T15:58:46.124 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1564: wait_for_clean: test 0 = 1 2019-12-21T15:58:46.125 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1565: wait_for_clean: test 0 '!=' -1 2019-12-21T15:58:46.125 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1566: wait_for_clean: loop=0 2019-12-21T15:58:46.125 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1567: wait_for_clean: num_active_clean=0 2019-12-21T15:58:46.125 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1575: wait_for_clean: eval 2019-12-21T15:58:46.125 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1576: wait_for_clean: sleep 0.1 2019-12-21T15:58:46.225 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1577: wait_for_clean: loop+=1 2019-12-21T15:58:46.225 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1559: wait_for_clean: true 2019-12-21T15:58:46.225 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1563: wait_for_clean: get_num_active_clean 2019-12-21T15:58:46.226 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: get_num_active_clean: local expression 2019-12-21T15:58:46.226 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2019-12-21T15:58:46.226 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("stale") | not)' 2019-12-21T15:58:46.226 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: ceph --format json pg dump pgs 2019-12-21T15:58:46.226 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2019-12-21T15:58:46.445 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1563: wait_for_clean: cur_active_clean=0 2019-12-21T15:58:46.445 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1564: wait_for_clean: get_num_pgs 2019-12-21T15:58:46.445 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: ceph --format json status 2019-12-21T15:58:46.446 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: jq .pgmap.num_pgs 2019-12-21T15:58:46.714 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1564: wait_for_clean: test 0 = 1 2019-12-21T15:58:46.715 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1565: wait_for_clean: test 0 '!=' 0 2019-12-21T15:58:46.715 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1568: wait_for_clean: get_is_making_recovery_progress 2019-12-21T15:58:46.715 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: get_is_making_recovery_progress: local recovery_progress 2019-12-21T15:58:46.715 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2019-12-21T15:58:46.715 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2019-12-21T15:58:46.715 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2019-12-21T15:58:46.716 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: ceph --format json status 2019-12-21T15:58:46.716 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2019-12-21T15:58:46.984 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: local progress=null 2019-12-21T15:58:46.984 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: test null '!=' null 2019-12-21T15:58:46.984 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1570: wait_for_clean: (( 1 >= 13 )) 2019-12-21T15:58:46.985 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1575: wait_for_clean: eval 2019-12-21T15:58:46.985 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1576: wait_for_clean: sleep 0.2 2019-12-21T15:58:47.185 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1577: wait_for_clean: loop+=1 2019-12-21T15:58:47.186 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1559: wait_for_clean: true 2019-12-21T15:58:47.188 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1563: wait_for_clean: get_num_active_clean 2019-12-21T15:58:47.188 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: get_num_active_clean: local expression 2019-12-21T15:58:47.188 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2019-12-21T15:58:47.188 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("stale") | not)' 2019-12-21T15:58:47.189 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: ceph --format json pg dump pgs 2019-12-21T15:58:47.189 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2019-12-21T15:58:47.407 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1563: wait_for_clean: cur_active_clean=0 2019-12-21T15:58:47.407 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1564: wait_for_clean: get_num_pgs 2019-12-21T15:58:47.407 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: ceph --format json status 2019-12-21T15:58:47.408 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: jq .pgmap.num_pgs 2019-12-21T15:58:47.677 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1564: wait_for_clean: test 0 = 1 2019-12-21T15:58:47.677 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1565: wait_for_clean: test 0 '!=' 0 2019-12-21T15:58:47.677 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1568: wait_for_clean: get_is_making_recovery_progress 2019-12-21T15:58:47.677 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: get_is_making_recovery_progress: local recovery_progress 2019-12-21T15:58:47.677 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2019-12-21T15:58:47.677 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2019-12-21T15:58:47.678 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2019-12-21T15:58:47.678 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: ceph --format json status 2019-12-21T15:58:47.678 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2019-12-21T15:58:47.969 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: local progress=null 2019-12-21T15:58:47.970 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: test null '!=' null 2019-12-21T15:58:47.970 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1570: wait_for_clean: (( 2 >= 13 )) 2019-12-21T15:58:47.970 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1575: wait_for_clean: eval 2019-12-21T15:58:47.970 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1576: wait_for_clean: sleep 0.4 2019-12-21T15:58:48.371 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1577: wait_for_clean: loop+=1 2019-12-21T15:58:48.371 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1559: wait_for_clean: true 2019-12-21T15:58:48.371 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1563: wait_for_clean: get_num_active_clean 2019-12-21T15:58:48.371 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: get_num_active_clean: local expression 2019-12-21T15:58:48.372 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2019-12-21T15:58:48.372 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("stale") | not)' 2019-12-21T15:58:48.372 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: ceph --format json pg dump pgs 2019-12-21T15:58:48.372 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2019-12-21T15:58:48.593 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1563: wait_for_clean: cur_active_clean=1 2019-12-21T15:58:48.593 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1564: wait_for_clean: get_num_pgs 2019-12-21T15:58:48.594 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: ceph --format json status 2019-12-21T15:58:48.594 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: jq .pgmap.num_pgs 2019-12-21T15:58:48.862 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1564: wait_for_clean: test 1 = 1 2019-12-21T15:58:48.862 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1564: wait_for_clean: break 2019-12-21T15:58:48.862 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: return 0 2019-12-21T15:58:48.863 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:373: TEST_auto_repair_bluestore_basic: get_not_primary testpool SOMETHING 2019-12-21T15:58:48.864 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1165: get_not_primary: local poolname=testpool 2019-12-21T15:58:48.864 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1166: get_not_primary: local objectname=SOMETHING 2019-12-21T15:58:48.864 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: get_not_primary: get_primary testpool SOMETHING 2019-12-21T15:58:48.864 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_primary: local poolname=testpool 2019-12-21T15:58:48.864 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1133: get_primary: local objectname=SOMETHING 2019-12-21T15:58:48.864 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1135: get_primary: ceph --format json osd map testpool SOMETHING 2019-12-21T15:58:48.864 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1136: get_primary: jq .acting_primary 2019-12-21T15:58:49.094 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: get_not_primary: local primary=1 2019-12-21T15:58:49.094 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1169: get_not_primary: ceph --format json osd map testpool SOMETHING 2019-12-21T15:58:49.094 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1170: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2019-12-21T15:58:49.324 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:373: TEST_auto_repair_bluestore_basic: objectstore_tool td/osd-scrub-repair 0 SOMETHING get-bytes td/osd-scrub-repair/COPY 2019-12-21T15:58:49.324 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: objectstore_tool: local dir=td/osd-scrub-repair 2019-12-21T15:58:49.324 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: objectstore_tool: shift 2019-12-21T15:58:49.324 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1235: objectstore_tool: local id=0 2019-12-21T15:58:49.325 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1236: objectstore_tool: shift 2019-12-21T15:58:49.325 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1238: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING get-bytes td/osd-scrub-repair/COPY 2019-12-21T15:58:49.325 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1205: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2019-12-21T15:58:49.325 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nowait: shift 2019-12-21T15:58:49.325 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1207: _objectstore_tool_nowait: local id=0 2019-12-21T15:58:49.325 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1208: _objectstore_tool_nowait: shift 2019-12-21T15:58:49.326 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1210: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2019-12-21T15:58:49.326 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2019-12-21T15:58:49.326 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2019-12-21T15:58:49.326 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2019-12-21T15:58:49.326 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2019-12-21T15:58:49.326 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2019-12-21T15:58:49.429 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2019-12-21T15:58:49.429 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1212: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING get-bytes td/osd-scrub-repair/COPY 2019-12-21T15:58:49.429 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1193: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2019-12-21T15:58:49.430 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1194: _objectstore_tool_nodown: shift 2019-12-21T15:58:49.430 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1195: _objectstore_tool_nodown: local id=0 2019-12-21T15:58:49.430 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: _objectstore_tool_nodown: shift 2019-12-21T15:58:49.430 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2019-12-21T15:58:49.430 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING get-bytes td/osd-scrub-repair/COPY 2019-12-21T15:58:50.589 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1213: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2019-12-21T15:58:50.590 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:842: activate_osd: local dir=td/osd-scrub-repair 2019-12-21T15:58:50.590 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: shift 2019-12-21T15:58:50.590 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: local id=0 2019-12-21T15:58:50.590 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: shift 2019-12-21T15:58:50.590 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local osd_data=td/osd-scrub-repair/0 2019-12-21T15:58:50.590 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local 'ceph_args=--fsid=973e3fc1-e63e-4bde-9db6-84c68fd5e93e --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2019-12-21T15:58:50.591 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2019-12-21T15:58:50.591 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-journal-size=100' 2019-12-21T15:58:50.591 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2019-12-21T15:58:50.591 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2019-12-21T15:58:50.591 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2019-12-21T15:58:50.591 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --chdir=' 2019-12-21T15:58:50.591 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+= 2019-12-21T15:58:50.592 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2019-12-21T15:58:50.592 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: get_asok_path 2019-12-21T15:58:50.592 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2019-12-21T15:58:50.592 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2019-12-21T15:58:50.592 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2019-12-21T15:58:50.592 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:58:50.592 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.119575 2019-12-21T15:58:50.593 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.119575/$cluster-$name.asok' 2019-12-21T15:58:50.593 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.119575/$cluster-$name.asok' 2019-12-21T15:58:50.593 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --debug-osd=20' 2019-12-21T15:58:50.593 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2019-12-21T15:58:50.593 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2019-12-21T15:58:50.593 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2019-12-21T15:58:50.594 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2019-12-21T15:58:50.594 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2019-12-21T15:58:50.594 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' ' 2019-12-21T15:58:50.594 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2019-12-21T15:58:50.594 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: mkdir -p td/osd-scrub-repair/0 2019-12-21T15:58:50.594 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: echo start osd.0 2019-12-21T15:58:50.594 INFO:tasks.workunit.client.0.smithi172.stderr:start osd.0 2019-12-21T15:58:50.595 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph-osd -i 0 --fsid=973e3fc1-e63e-4bde-9db6-84c68fd5e93e --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.119575/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2019-12-21T15:58:50.595 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: cat td/osd-scrub-repair/0/whoami 2019-12-21T15:58:50.596 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: '[' 0 = 0 ']' 2019-12-21T15:58:50.598 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph osd dump --format=json 2019-12-21T15:58:50.598 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: jq '.flags_set[]' 2019-12-21T15:58:50.598 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: grep -q '"noup"' 2019-12-21T15:58:50.613 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:58:50.611+0000 7f5d1a7c8ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T15:58:50.614 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:58:50.613+0000 7f5d1a7c8ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T15:58:50.616 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:58:50.615+0000 7f5d1a7c8ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T15:58:50.830 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: wait_for_osd up 0 2019-12-21T15:58:50.830 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:914: wait_for_osd: local state=up 2019-12-21T15:58:50.831 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local id=0 2019-12-21T15:58:50.831 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: status=1 2019-12-21T15:58:50.831 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i=0 )) 2019-12-21T15:58:50.831 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i < 300 )) 2019-12-21T15:58:50.831 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: echo 0 2019-12-21T15:58:50.831 INFO:tasks.workunit.client.0.smithi172.stderr:0 2019-12-21T15:58:50.831 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: ceph osd dump 2019-12-21T15:58:50.832 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: grep 'osd.0 up' 2019-12-21T15:58:51.056 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: sleep 1 2019-12-21T15:58:51.132 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:58:51.130+0000 7f5d1a7c8ec0 -1 Falling back to public interface 2019-12-21T15:58:51.869 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:58:51.867+0000 7f5d1a7c8ec0 -1 osd.0 29 log_to_monitors {default=true} 2019-12-21T15:58:52.058 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i++ )) 2019-12-21T15:58:52.058 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i < 300 )) 2019-12-21T15:58:52.059 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: echo 1 2019-12-21T15:58:52.059 INFO:tasks.workunit.client.0.smithi172.stderr:1 2019-12-21T15:58:52.059 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: ceph osd dump 2019-12-21T15:58:52.059 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: grep 'osd.0 up' 2019-12-21T15:58:52.294 INFO:tasks.workunit.client.0.smithi172.stderr:osd.0 up in weight 1 up_from 32 up_thru 0 down_at 30 last_clean_interval [28,29) [v2:127.0.0.1:6802/126890,v1:127.0.0.1:6803/126890] [v2:127.0.0.1:6804/126890,v1:127.0.0.1:6805/126890] exists,up 1ae93a90-3644-4907-a6e4-08b0470a2899 2019-12-21T15:58:52.294 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=0 2019-12-21T15:58:52.294 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: break 2019-12-21T15:58:52.294 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: return 0 2019-12-21T15:58:52.294 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1239: objectstore_tool: wait_for_clean 2019-12-21T15:58:52.294 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1548: wait_for_clean: local cmd= 2019-12-21T15:58:52.295 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1549: wait_for_clean: local num_active_clean=-1 2019-12-21T15:58:52.295 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1550: wait_for_clean: local cur_active_clean 2019-12-21T15:58:52.295 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1551: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2019-12-21T15:58:52.295 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1551: wait_for_clean: get_timeout_delays 90 .1 2019-12-21T15:58:52.295 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1494: get_timeout_delays: shopt -q -o xtrace 2019-12-21T15:58:52.295 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1494: get_timeout_delays: echo true 2019-12-21T15:58:52.295 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1494: get_timeout_delays: local trace=true 2019-12-21T15:58:52.296 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1495: get_timeout_delays: true 2019-12-21T15:58:52.296 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1495: get_timeout_delays: shopt -u -o xtrace 2019-12-21T15:58:52.385 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1551: wait_for_clean: local -a delays 2019-12-21T15:58:52.386 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1552: wait_for_clean: local -i loop=0 2019-12-21T15:58:52.386 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1554: wait_for_clean: flush_pg_stats 2019-12-21T15:58:52.386 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1977: flush_pg_stats: local timeout=300 2019-12-21T15:58:52.386 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1979: flush_pg_stats: ceph osd ls 2019-12-21T15:58:52.613 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1979: flush_pg_stats: ids='0 2019-12-21T15:58:52.613 INFO:tasks.workunit.client.0.smithi172.stderr:1 2019-12-21T15:58:52.614 INFO:tasks.workunit.client.0.smithi172.stderr:2' 2019-12-21T15:58:52.614 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1980: flush_pg_stats: seqs= 2019-12-21T15:58:52.614 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: flush_pg_stats: for osd in $ids 2019-12-21T15:58:52.614 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2019-12-21T15:58:52.733 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: seq=137438953474 2019-12-21T15:58:52.735 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: flush_pg_stats: seqs=' 0-137438953474' 2019-12-21T15:58:52.735 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: flush_pg_stats: for osd in $ids 2019-12-21T15:58:52.736 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2019-12-21T15:58:52.843 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: seq=42949672975 2019-12-21T15:58:52.843 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: flush_pg_stats: seqs=' 0-137438953474 1-42949672975' 2019-12-21T15:58:52.843 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: flush_pg_stats: for osd in $ids 2019-12-21T15:58:52.843 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2019-12-21T15:58:52.954 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: seq=64424509453 2019-12-21T15:58:52.954 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: flush_pg_stats: seqs=' 0-137438953474 1-42949672975 2-64424509453' 2019-12-21T15:58:52.954 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: flush_pg_stats: for s in $seqs 2019-12-21T15:58:52.955 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: echo 0-137438953474 2019-12-21T15:58:52.955 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: cut -d - -f 1 2019-12-21T15:58:52.955 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: osd=0 2019-12-21T15:58:52.956 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: echo 0-137438953474 2019-12-21T15:58:52.956 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: cut -d - -f 2 2019-12-21T15:58:52.957 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: seq=137438953474 2019-12-21T15:58:52.957 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1989: flush_pg_stats: echo 'waiting osd.0 seq 137438953474' 2019-12-21T15:58:52.957 INFO:tasks.workunit.client.0.smithi172.stderr:waiting osd.0 seq 137438953474 2019-12-21T15:58:52.957 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 0 2019-12-21T15:58:53.182 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 120259084291 -lt 137438953474 2019-12-21T15:58:53.182 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1991: flush_pg_stats: sleep 1 2019-12-21T15:58:54.184 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1992: flush_pg_stats: '[' 300 -eq 0 ']' 2019-12-21T15:58:54.184 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 0 2019-12-21T15:58:54.415 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 137438953474 -lt 137438953474 2019-12-21T15:58:54.415 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: flush_pg_stats: for s in $seqs 2019-12-21T15:58:54.416 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: echo 1-42949672975 2019-12-21T15:58:54.417 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: cut -d - -f 1 2019-12-21T15:58:54.417 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: osd=1 2019-12-21T15:58:54.418 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: echo 1-42949672975 2019-12-21T15:58:54.419 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: cut -d - -f 2 2019-12-21T15:58:54.419 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: seq=42949672975 2019-12-21T15:58:54.419 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1989: flush_pg_stats: echo 'waiting osd.1 seq 42949672975' 2019-12-21T15:58:54.419 INFO:tasks.workunit.client.0.smithi172.stderr:waiting osd.1 seq 42949672975 2019-12-21T15:58:54.420 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 1 2019-12-21T15:58:54.647 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 42949672975 -lt 42949672975 2019-12-21T15:58:54.648 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: flush_pg_stats: for s in $seqs 2019-12-21T15:58:54.648 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: echo 2-64424509453 2019-12-21T15:58:54.649 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: cut -d - -f 1 2019-12-21T15:58:54.649 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: osd=2 2019-12-21T15:58:54.649 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: echo 2-64424509453 2019-12-21T15:58:54.650 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: cut -d - -f 2 2019-12-21T15:58:54.651 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: seq=64424509453 2019-12-21T15:58:54.651 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1989: flush_pg_stats: echo 'waiting osd.2 seq 64424509453' 2019-12-21T15:58:54.651 INFO:tasks.workunit.client.0.smithi172.stderr:waiting osd.2 seq 64424509453 2019-12-21T15:58:54.651 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 2 2019-12-21T15:58:54.878 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 64424509453 -lt 64424509453 2019-12-21T15:58:54.879 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1555: wait_for_clean: get_num_pgs 2019-12-21T15:58:54.879 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: ceph --format json status 2019-12-21T15:58:54.880 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: jq .pgmap.num_pgs 2019-12-21T15:58:55.149 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1555: wait_for_clean: test 1 == 0 2019-12-21T15:58:55.149 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1559: wait_for_clean: true 2019-12-21T15:58:55.149 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1563: wait_for_clean: get_num_active_clean 2019-12-21T15:58:55.149 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: get_num_active_clean: local expression 2019-12-21T15:58:55.150 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2019-12-21T15:58:55.150 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("stale") | not)' 2019-12-21T15:58:55.150 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: ceph --format json pg dump pgs 2019-12-21T15:58:55.150 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2019-12-21T15:58:55.370 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1563: wait_for_clean: cur_active_clean=1 2019-12-21T15:58:55.371 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1564: wait_for_clean: get_num_pgs 2019-12-21T15:58:55.371 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: ceph --format json status 2019-12-21T15:58:55.372 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: jq .pgmap.num_pgs 2019-12-21T15:58:55.640 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1564: wait_for_clean: test 1 = 1 2019-12-21T15:58:55.641 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1564: wait_for_clean: break 2019-12-21T15:58:55.641 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: return 0 2019-12-21T15:58:55.641 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:374: TEST_auto_repair_bluestore_basic: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2019-12-21T15:58:55.641 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:375: TEST_auto_repair_bluestore_basic: grep scrub_finish td/osd-scrub-repair/osd.1.log 2019-12-21T15:58:55.643 INFO:tasks.workunit.client.0.smithi172.stdout:2019-12-21T15:58:28.891+0000 7f1447be4700 20 osd.1 pg_epoch: 25 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=24/25 n=1 ec=17/17 lis/c=24/24 les/c/f=25/25/0 sis=24) [1,0] r=0 lpr=24 crt=21'1 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+repair AUTO_REPAIR] scrub_finish 2019-12-21T15:58:55.643 INFO:tasks.workunit.client.0.smithi172.stdout:2019-12-21T15:58:28.891+0000 7f1447be4700 20 osd.1 pg_epoch: 25 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=24/25 n=1 ec=17/17 lis/c=24/24 les/c/f=25/25/0 sis=24) [1,0] r=0 lpr=24 crt=21'1 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair AUTO_REPAIR mbc={255={(1+0)=1}}] scrub_finish All may be fixed 2019-12-21T15:58:55.644 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:378: TEST_auto_repair_bluestore_basic: teardown td/osd-scrub-repair 2019-12-21T15:58:55.644 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2019-12-21T15:58:55.644 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2019-12-21T15:58:55.644 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2019-12-21T15:58:55.644 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2019-12-21T15:58:55.644 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2019-12-21T15:58:55.644 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2019-12-21T15:58:55.645 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2019-12-21T15:58:55.645 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2019-12-21T15:58:55.750 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2019-12-21T15:58:55.750 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2019-12-21T15:58:55.751 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2019-12-21T15:58:55.751 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2019-12-21T15:58:55.752 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2019-12-21T15:58:55.752 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2019-12-21T15:58:55.752 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2019-12-21T15:58:55.754 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local 'pattern=|/usr/lib/systemd/systemd-coredump %P %u %g %s %t %c %h %e' 2019-12-21T15:58:55.754 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' '|' = '|' ']' 2019-12-21T15:58:55.755 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:177: teardown: pattern= 2019-12-21T15:58:55.755 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2019-12-21T15:58:55.755 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname '' 2019-12-21T15:58:55.755 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls . 2019-12-21T15:58:55.756 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2019-12-21T15:58:55.756 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2019-12-21T15:58:55.762 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2019-12-21T15:58:55.762 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:58:55.763 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.119575 2019-12-21T15:58:55.763 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.119575 2019-12-21T15:58:55.765 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2019-12-21T15:58:55.765 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2019-12-21T15:58:55.765 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:62: run: for func in $funcs 2019-12-21T15:58:55.765 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: TEST_auto_repair_bluestore_failed td/osd-scrub-repair 2019-12-21T15:58:55.765 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:437: TEST_auto_repair_bluestore_failed: local dir=td/osd-scrub-repair 2019-12-21T15:58:55.766 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:438: TEST_auto_repair_bluestore_failed: local poolname=testpool 2019-12-21T15:58:55.766 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:441: TEST_auto_repair_bluestore_failed: setup td/osd-scrub-repair 2019-12-21T15:58:55.766 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2019-12-21T15:58:55.766 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2019-12-21T15:58:55.766 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2019-12-21T15:58:55.767 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2019-12-21T15:58:55.767 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2019-12-21T15:58:55.767 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2019-12-21T15:58:55.767 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2019-12-21T15:58:55.767 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2019-12-21T15:58:55.767 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2019-12-21T15:58:55.768 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2019-12-21T15:58:55.769 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2019-12-21T15:58:55.769 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2019-12-21T15:58:55.769 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2019-12-21T15:58:55.769 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2019-12-21T15:58:55.769 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2019-12-21T15:58:55.769 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2019-12-21T15:58:55.770 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2019-12-21T15:58:55.771 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local 'pattern=|/usr/lib/systemd/systemd-coredump %P %u %g %s %t %c %h %e' 2019-12-21T15:58:55.771 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' '|' = '|' ']' 2019-12-21T15:58:55.771 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:177: teardown: pattern= 2019-12-21T15:58:55.771 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2019-12-21T15:58:55.771 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname '' 2019-12-21T15:58:55.771 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls . 2019-12-21T15:58:55.772 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2019-12-21T15:58:55.772 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2019-12-21T15:58:55.773 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2019-12-21T15:58:55.773 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:58:55.773 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.119575 2019-12-21T15:58:55.773 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.119575 2019-12-21T15:58:55.774 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2019-12-21T15:58:55.774 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2019-12-21T15:58:55.774 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2019-12-21T15:58:55.775 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2019-12-21T15:58:55.775 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:58:55.775 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.119575 2019-12-21T15:58:55.775 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.119575 2019-12-21T15:58:55.777 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2019-12-21T15:58:55.777 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2019-12-21T15:58:55.777 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2019-12-21T15:58:55.777 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2019-12-21T15:58:55.777 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:442: TEST_auto_repair_bluestore_failed: run_mon td/osd-scrub-repair a 2019-12-21T15:58:55.778 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2019-12-21T15:58:55.778 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2019-12-21T15:58:55.778 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2019-12-21T15:58:55.778 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2019-12-21T15:58:55.778 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2019-12-21T15:58:55.778 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2019-12-21T15:58:56.063 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2019-12-21T15:58:56.064 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2019-12-21T15:58:56.064 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2019-12-21T15:58:56.064 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2019-12-21T15:58:56.064 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:58:56.064 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.119575 2019-12-21T15:58:56.065 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.119575/$cluster-$name.asok' 2019-12-21T15:58:56.065 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.119575/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 2019-12-21T15:58:56.254 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: cat 2019-12-21T15:58:56.257 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: get_config mon a fsid 2019-12-21T15:58:56.257 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: get_config: local daemon=mon 2019-12-21T15:58:56.257 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: get_config: local id=a 2019-12-21T15:58:56.257 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local config=fsid 2019-12-21T15:58:56.257 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: jq -r .fsid 2019-12-21T15:58:56.258 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: get_asok_path mon.a 2019-12-21T15:58:56.258 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2019-12-21T15:58:56.258 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2019-12-21T15:58:56.258 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:58:56.258 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:58:56.258 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.119575 2019-12-21T15:58:56.258 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.119575/ceph-mon.a.asok 2019-12-21T15:58:56.259 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: CEPH_ARGS= 2019-12-21T15:58:56.259 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: ceph --format json daemon /tmp/ceph-asok.119575/ceph-mon.a.asok config get fsid 2019-12-21T15:58:56.343 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: get_config mon a mon_host 2019-12-21T15:58:56.343 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: get_config: local daemon=mon 2019-12-21T15:58:56.343 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: get_config: local id=a 2019-12-21T15:58:56.344 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local config=mon_host 2019-12-21T15:58:56.344 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: jq -r .mon_host 2019-12-21T15:58:56.344 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: get_asok_path mon.a 2019-12-21T15:58:56.344 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2019-12-21T15:58:56.344 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2019-12-21T15:58:56.344 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T15:58:56.344 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:58:56.345 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.119575 2019-12-21T15:58:56.345 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.119575/ceph-mon.a.asok 2019-12-21T15:58:56.345 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: CEPH_ARGS= 2019-12-21T15:58:56.345 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: ceph --format json daemon /tmp/ceph-asok.119575/ceph-mon.a.asok config get mon_host 2019-12-21T15:58:56.432 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:443: TEST_auto_repair_bluestore_failed: run_mgr td/osd-scrub-repair x 2019-12-21T15:58:56.432 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:551: run_mgr: local dir=td/osd-scrub-repair 2019-12-21T15:58:56.433 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: shift 2019-12-21T15:58:56.434 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local id=x 2019-12-21T15:58:56.434 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2019-12-21T15:58:56.434 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local data=td/osd-scrub-repair/x 2019-12-21T15:58:56.434 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:558: run_mgr: get_asok_path 2019-12-21T15:58:56.434 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2019-12-21T15:58:56.434 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2019-12-21T15:58:56.434 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2019-12-21T15:58:56.439 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:58:56.439 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.119575 2019-12-21T15:58:56.439 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.119575/$cluster-$name.asok' 2019-12-21T15:58:56.439 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:558: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2019-12-21T15:58:56.439 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:558: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.119575/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2019-12-21T15:58:57.054 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:446: TEST_auto_repair_bluestore_failed: local 'ceph_osd_args=--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2019-12-21T15:58:57.054 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:447: TEST_auto_repair_bluestore_failed: seq 0 2 2019-12-21T15:58:57.059 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:447: TEST_auto_repair_bluestore_failed: for id in $(seq 0 2) 2019-12-21T15:58:57.059 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:448: TEST_auto_repair_bluestore_failed: run_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2019-12-21T15:58:57.059 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local dir=td/osd-scrub-repair 2019-12-21T15:58:57.059 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: shift 2019-12-21T15:58:57.059 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local id=0 2019-12-21T15:58:57.059 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2019-12-21T15:58:57.060 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local osd_data=td/osd-scrub-repair/0 2019-12-21T15:58:57.060 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local 'ceph_args=--fsid=973e3fc1-e63e-4bde-9db6-84c68fd5e93e --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2019-12-21T15:58:57.060 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2019-12-21T15:58:57.060 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-journal-size=100' 2019-12-21T15:58:57.061 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2019-12-21T15:58:57.061 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2019-12-21T15:58:57.061 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2019-12-21T15:58:57.061 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --chdir=' 2019-12-21T15:58:57.061 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+= 2019-12-21T15:58:57.061 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2019-12-21T15:58:57.061 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: get_asok_path 2019-12-21T15:58:57.062 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2019-12-21T15:58:57.062 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2019-12-21T15:58:57.062 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2019-12-21T15:58:57.062 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:58:57.062 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.119575 2019-12-21T15:58:57.062 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.119575/$cluster-$name.asok' 2019-12-21T15:58:57.062 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.119575/$cluster-$name.asok' 2019-12-21T15:58:57.063 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --debug-osd=20' 2019-12-21T15:58:57.063 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-ms=1' 2019-12-21T15:58:57.063 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-monc=20' 2019-12-21T15:58:57.063 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2019-12-21T15:58:57.063 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2019-12-21T15:58:57.063 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2019-12-21T15:58:57.063 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2019-12-21T15:58:57.064 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2019-12-21T15:58:57.064 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' ' 2019-12-21T15:58:57.064 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2019-12-21T15:58:57.064 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: mkdir -p td/osd-scrub-repair/0 2019-12-21T15:58:57.064 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: uuidgen 2019-12-21T15:58:57.064 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: local uuid=4236b440-d79c-4b13-9ee0-6323732a2f6e 2019-12-21T15:58:57.064 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: echo 'add osd0 4236b440-d79c-4b13-9ee0-6323732a2f6e' 2019-12-21T15:58:57.065 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: ceph-authtool --gen-print-key 2019-12-21T15:58:57.065 INFO:tasks.workunit.client.0.smithi172.stdout:add osd0 4236b440-d79c-4b13-9ee0-6323732a2f6e 2019-12-21T15:58:57.073 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: OSD_SECRET=AQBBQf5dfOUwBBAAT1uW+aBD9KN5rffm3FMRzw== 2019-12-21T15:58:57.073 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo '{"cephx_secret": "AQBBQf5dfOUwBBAAT1uW+aBD9KN5rffm3FMRzw=="}' 2019-12-21T15:58:57.073 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph osd new 4236b440-d79c-4b13-9ee0-6323732a2f6e -i td/osd-scrub-repair/0/new.json 2019-12-21T15:58:57.337 INFO:tasks.workunit.client.0.smithi172.stdout:0 2019-12-21T15:58:57.347 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: rm td/osd-scrub-repair/0/new.json 2019-12-21T15:58:57.348 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-osd -i 0 --fsid=973e3fc1-e63e-4bde-9db6-84c68fd5e93e --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.119575/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQBBQf5dfOUwBBAAT1uW+aBD9KN5rffm3FMRzw== --osd-uuid 4236b440-d79c-4b13-9ee0-6323732a2f6e 2019-12-21T15:58:57.371 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:58:57.369+0000 7f7098415ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T15:58:57.372 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:58:57.371+0000 7f7098415ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T15:58:57.373 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:58:57.372+0000 7f7098415ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T15:58:57.374 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:58:57.372+0000 7f7098415ec0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2019-12-21T15:58:57.374 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:58:57.372+0000 7f7098415ec0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2019-12-21T15:58:57.374 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:58:57.372+0000 7f7098415ec0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2019-12-21T15:58:57.428 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:58:57.427+0000 7f7098415ec0 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2019-12-21T15:59:00.473 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2019-12-21T15:59:00.473 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: cat 2019-12-21T15:59:00.474 INFO:tasks.workunit.client.0.smithi172.stdout:adding osd0 key to auth repository 2019-12-21T15:59:00.474 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: echo adding osd0 key to auth repository 2019-12-21T15:59:00.474 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2019-12-21T15:59:00.671 INFO:tasks.workunit.client.0.smithi172.stdout:start osd.0 2019-12-21T15:59:00.672 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo start osd.0 2019-12-21T15:59:00.672 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph osd dump --format=json 2019-12-21T15:59:00.672 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: jq '.flags_set[]' 2019-12-21T15:59:00.672 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: grep -q '"noup"' 2019-12-21T15:59:00.673 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph-osd -i 0 --fsid=973e3fc1-e63e-4bde-9db6-84c68fd5e93e --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.119575/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2019-12-21T15:59:00.693 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:59:00.691+0000 7ff712a3eec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T15:59:00.694 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:59:00.693+0000 7ff712a3eec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T15:59:00.697 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:59:00.695+0000 7ff712a3eec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T15:59:00.903 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: wait_for_osd up 0 2019-12-21T15:59:00.903 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:914: wait_for_osd: local state=up 2019-12-21T15:59:00.903 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local id=0 2019-12-21T15:59:00.903 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: status=1 2019-12-21T15:59:00.904 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i=0 )) 2019-12-21T15:59:00.904 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i < 300 )) 2019-12-21T15:59:00.904 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: echo 0 2019-12-21T15:59:00.904 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: ceph osd dump 2019-12-21T15:59:00.904 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: grep 'osd.0 up' 2019-12-21T15:59:00.904 INFO:tasks.workunit.client.0.smithi172.stdout:0 2019-12-21T15:59:01.130 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: sleep 1 2019-12-21T15:59:01.212 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:59:01.210+0000 7ff712a3eec0 -1 Falling back to public interface 2019-12-21T15:59:01.935 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:59:01.934+0000 7ff712a3eec0 -1 osd.0 0 log_to_monitors {default=true} 2019-12-21T15:59:02.131 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i++ )) 2019-12-21T15:59:02.131 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i < 300 )) 2019-12-21T15:59:02.131 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: echo 1 2019-12-21T15:59:02.132 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: ceph osd dump 2019-12-21T15:59:02.132 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: grep 'osd.0 up' 2019-12-21T15:59:02.132 INFO:tasks.workunit.client.0.smithi172.stdout:1 2019-12-21T15:59:02.416 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: sleep 1 2019-12-21T15:59:03.418 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i++ )) 2019-12-21T15:59:03.418 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i < 300 )) 2019-12-21T15:59:03.419 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: echo 2 2019-12-21T15:59:03.419 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: ceph osd dump 2019-12-21T15:59:03.419 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: grep 'osd.0 up' 2019-12-21T15:59:03.419 INFO:tasks.workunit.client.0.smithi172.stdout:2 2019-12-21T15:59:03.653 INFO:tasks.workunit.client.0.smithi172.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/127871,v1:127.0.0.1:6803/127871] [v2:127.0.0.1:6804/127871,v1:127.0.0.1:6805/127871] exists,up 4236b440-d79c-4b13-9ee0-6323732a2f6e 2019-12-21T15:59:03.654 INFO:tasks.workunit.client.0.smithi172.stdout:add osd1 9bb8026b-4cfb-486b-aa07-b00acd8d49cd 2019-12-21T15:59:03.654 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=0 2019-12-21T15:59:03.654 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: break 2019-12-21T15:59:03.654 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: return 0 2019-12-21T15:59:03.654 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:447: TEST_auto_repair_bluestore_failed: for id in $(seq 0 2) 2019-12-21T15:59:03.654 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:448: TEST_auto_repair_bluestore_failed: run_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2019-12-21T15:59:03.655 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local dir=td/osd-scrub-repair 2019-12-21T15:59:03.655 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: shift 2019-12-21T15:59:03.655 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local id=1 2019-12-21T15:59:03.655 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2019-12-21T15:59:03.655 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local osd_data=td/osd-scrub-repair/1 2019-12-21T15:59:03.655 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local 'ceph_args=--fsid=973e3fc1-e63e-4bde-9db6-84c68fd5e93e --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2019-12-21T15:59:03.655 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2019-12-21T15:59:03.656 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-journal-size=100' 2019-12-21T15:59:03.656 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2019-12-21T15:59:03.656 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2019-12-21T15:59:03.656 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2019-12-21T15:59:03.657 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --chdir=' 2019-12-21T15:59:03.657 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+= 2019-12-21T15:59:03.657 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2019-12-21T15:59:03.657 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: get_asok_path 2019-12-21T15:59:03.657 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2019-12-21T15:59:03.657 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2019-12-21T15:59:03.657 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2019-12-21T15:59:03.658 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:59:03.658 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.119575 2019-12-21T15:59:03.658 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.119575/$cluster-$name.asok' 2019-12-21T15:59:03.658 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.119575/$cluster-$name.asok' 2019-12-21T15:59:03.658 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --debug-osd=20' 2019-12-21T15:59:03.658 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-ms=1' 2019-12-21T15:59:03.658 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-monc=20' 2019-12-21T15:59:03.659 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2019-12-21T15:59:03.659 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2019-12-21T15:59:03.659 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2019-12-21T15:59:03.659 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2019-12-21T15:59:03.659 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2019-12-21T15:59:03.659 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' ' 2019-12-21T15:59:03.659 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2019-12-21T15:59:03.660 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: mkdir -p td/osd-scrub-repair/1 2019-12-21T15:59:03.660 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: uuidgen 2019-12-21T15:59:03.660 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: local uuid=9bb8026b-4cfb-486b-aa07-b00acd8d49cd 2019-12-21T15:59:03.660 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: echo 'add osd1 9bb8026b-4cfb-486b-aa07-b00acd8d49cd' 2019-12-21T15:59:03.660 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: ceph-authtool --gen-print-key 2019-12-21T15:59:03.666 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: OSD_SECRET=AQBHQf5dvtibJxAA7fN8Xj/Vnq4BGTaRmvklBA== 2019-12-21T15:59:03.667 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo '{"cephx_secret": "AQBHQf5dvtibJxAA7fN8Xj/Vnq4BGTaRmvklBA=="}' 2019-12-21T15:59:03.667 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph osd new 9bb8026b-4cfb-486b-aa07-b00acd8d49cd -i td/osd-scrub-repair/1/new.json 2019-12-21T15:59:03.993 INFO:tasks.workunit.client.0.smithi172.stdout:1 2019-12-21T15:59:04.003 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: rm td/osd-scrub-repair/1/new.json 2019-12-21T15:59:04.004 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-osd -i 1 --fsid=973e3fc1-e63e-4bde-9db6-84c68fd5e93e --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.119575/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQBHQf5dvtibJxAA7fN8Xj/Vnq4BGTaRmvklBA== --osd-uuid 9bb8026b-4cfb-486b-aa07-b00acd8d49cd 2019-12-21T15:59:04.026 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:59:04.025+0000 7f694cb31ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T15:59:04.028 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:59:04.027+0000 7f694cb31ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T15:59:04.030 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:59:04.028+0000 7f694cb31ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T15:59:04.030 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:59:04.029+0000 7f694cb31ec0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2019-12-21T15:59:04.031 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:59:04.029+0000 7f694cb31ec0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2019-12-21T15:59:04.031 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:59:04.029+0000 7f694cb31ec0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2019-12-21T15:59:04.084 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:59:04.083+0000 7f694cb31ec0 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2019-12-21T15:59:07.039 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2019-12-21T15:59:07.039 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: cat 2019-12-21T15:59:07.040 INFO:tasks.workunit.client.0.smithi172.stdout:adding osd1 key to auth repository 2019-12-21T15:59:07.040 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: echo adding osd1 key to auth repository 2019-12-21T15:59:07.040 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2019-12-21T15:59:07.334 INFO:tasks.workunit.client.0.smithi172.stdout:start osd.1 2019-12-21T15:59:07.334 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo start osd.1 2019-12-21T15:59:07.335 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph osd dump --format=json 2019-12-21T15:59:07.335 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph-osd -i 1 --fsid=973e3fc1-e63e-4bde-9db6-84c68fd5e93e --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.119575/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2019-12-21T15:59:07.335 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: jq '.flags_set[]' 2019-12-21T15:59:07.336 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: grep -q '"noup"' 2019-12-21T15:59:07.354 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:59:07.352+0000 7f8ffb81eec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T15:59:07.356 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:59:07.354+0000 7f8ffb81eec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T15:59:07.358 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:59:07.355+0000 7f8ffb81eec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T15:59:07.563 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: wait_for_osd up 1 2019-12-21T15:59:07.563 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:914: wait_for_osd: local state=up 2019-12-21T15:59:07.563 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local id=1 2019-12-21T15:59:07.564 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: status=1 2019-12-21T15:59:07.564 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i=0 )) 2019-12-21T15:59:07.564 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i < 300 )) 2019-12-21T15:59:07.564 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: echo 0 2019-12-21T15:59:07.564 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: ceph osd dump 2019-12-21T15:59:07.564 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: grep 'osd.1 up' 2019-12-21T15:59:07.565 INFO:tasks.workunit.client.0.smithi172.stdout:0 2019-12-21T15:59:07.792 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: sleep 1 2019-12-21T15:59:07.877 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:59:07.875+0000 7f8ffb81eec0 -1 Falling back to public interface 2019-12-21T15:59:08.658 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:59:08.656+0000 7f8ffb81eec0 -1 osd.1 0 log_to_monitors {default=true} 2019-12-21T15:59:08.790 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i++ )) 2019-12-21T15:59:08.790 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i < 300 )) 2019-12-21T15:59:08.790 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: echo 1 2019-12-21T15:59:08.790 INFO:tasks.workunit.client.0.smithi172.stdout:1 2019-12-21T15:59:08.791 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: grep 'osd.1 up' 2019-12-21T15:59:08.791 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: ceph osd dump 2019-12-21T15:59:09.053 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: sleep 1 2019-12-21T15:59:10.054 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i++ )) 2019-12-21T15:59:10.055 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i < 300 )) 2019-12-21T15:59:10.055 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: echo 2 2019-12-21T15:59:10.055 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: ceph osd dump 2019-12-21T15:59:10.055 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: grep 'osd.1 up' 2019-12-21T15:59:10.055 INFO:tasks.workunit.client.0.smithi172.stdout:2 2019-12-21T15:59:10.292 INFO:tasks.workunit.client.0.smithi172.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/128248,v1:127.0.0.1:6811/128248] [v2:127.0.0.1:6812/128248,v1:127.0.0.1:6813/128248] exists,up 9bb8026b-4cfb-486b-aa07-b00acd8d49cd 2019-12-21T15:59:10.292 INFO:tasks.workunit.client.0.smithi172.stdout:add osd2 9ebd74a3-4d1d-4717-b3f8-baa706f9765d 2019-12-21T15:59:10.292 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=0 2019-12-21T15:59:10.293 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: break 2019-12-21T15:59:10.293 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: return 0 2019-12-21T15:59:10.293 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:447: TEST_auto_repair_bluestore_failed: for id in $(seq 0 2) 2019-12-21T15:59:10.293 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:448: TEST_auto_repair_bluestore_failed: run_osd td/osd-scrub-repair 2 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2019-12-21T15:59:10.293 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local dir=td/osd-scrub-repair 2019-12-21T15:59:10.293 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: shift 2019-12-21T15:59:10.293 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local id=2 2019-12-21T15:59:10.294 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2019-12-21T15:59:10.294 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local osd_data=td/osd-scrub-repair/2 2019-12-21T15:59:10.294 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local 'ceph_args=--fsid=973e3fc1-e63e-4bde-9db6-84c68fd5e93e --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2019-12-21T15:59:10.294 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2019-12-21T15:59:10.294 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-journal-size=100' 2019-12-21T15:59:10.295 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2019-12-21T15:59:10.295 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2019-12-21T15:59:10.295 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2019-12-21T15:59:10.295 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --chdir=' 2019-12-21T15:59:10.295 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+= 2019-12-21T15:59:10.296 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2019-12-21T15:59:10.296 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: get_asok_path 2019-12-21T15:59:10.301 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2019-12-21T15:59:10.302 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2019-12-21T15:59:10.302 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2019-12-21T15:59:10.302 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:59:10.302 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.119575 2019-12-21T15:59:10.303 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.119575/$cluster-$name.asok' 2019-12-21T15:59:10.303 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.119575/$cluster-$name.asok' 2019-12-21T15:59:10.303 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --debug-osd=20' 2019-12-21T15:59:10.303 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-ms=1' 2019-12-21T15:59:10.304 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-monc=20' 2019-12-21T15:59:10.304 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2019-12-21T15:59:10.304 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2019-12-21T15:59:10.304 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2019-12-21T15:59:10.304 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2019-12-21T15:59:10.305 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2019-12-21T15:59:10.305 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' ' 2019-12-21T15:59:10.305 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2019-12-21T15:59:10.305 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: mkdir -p td/osd-scrub-repair/2 2019-12-21T15:59:10.306 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: uuidgen 2019-12-21T15:59:10.306 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: local uuid=9ebd74a3-4d1d-4717-b3f8-baa706f9765d 2019-12-21T15:59:10.306 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: echo 'add osd2 9ebd74a3-4d1d-4717-b3f8-baa706f9765d' 2019-12-21T15:59:10.306 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: ceph-authtool --gen-print-key 2019-12-21T15:59:10.307 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: OSD_SECRET=AQBOQf5do+wLEhAAEgOHI91OC1AmmXQkl9syVg== 2019-12-21T15:59:10.307 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo '{"cephx_secret": "AQBOQf5do+wLEhAAEgOHI91OC1AmmXQkl9syVg=="}' 2019-12-21T15:59:10.308 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph osd new 9ebd74a3-4d1d-4717-b3f8-baa706f9765d -i td/osd-scrub-repair/2/new.json 2019-12-21T15:59:10.616 INFO:tasks.workunit.client.0.smithi172.stdout:2 2019-12-21T15:59:10.626 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: rm td/osd-scrub-repair/2/new.json 2019-12-21T15:59:10.627 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-osd -i 2 --fsid=973e3fc1-e63e-4bde-9db6-84c68fd5e93e --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.119575/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQBOQf5do+wLEhAAEgOHI91OC1AmmXQkl9syVg== --osd-uuid 9ebd74a3-4d1d-4717-b3f8-baa706f9765d 2019-12-21T15:59:10.649 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:59:10.646+0000 7f845aa8bec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T15:59:10.658 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:59:10.654+0000 7f845aa8bec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T15:59:10.663 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:59:10.656+0000 7f845aa8bec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T15:59:10.663 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:59:10.656+0000 7f845aa8bec0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2019-12-21T15:59:10.663 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:59:10.656+0000 7f845aa8bec0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2019-12-21T15:59:10.664 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:59:10.656+0000 7f845aa8bec0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2019-12-21T15:59:10.716 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:59:10.713+0000 7f845aa8bec0 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2019-12-21T15:59:13.699 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2019-12-21T15:59:13.700 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: cat 2019-12-21T15:59:13.701 INFO:tasks.workunit.client.0.smithi172.stdout:adding osd2 key to auth repository 2019-12-21T15:59:13.701 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: echo adding osd2 key to auth repository 2019-12-21T15:59:13.701 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2019-12-21T15:59:13.982 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo start osd.2 2019-12-21T15:59:13.982 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph-osd -i 2 --fsid=973e3fc1-e63e-4bde-9db6-84c68fd5e93e --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.119575/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2019-12-21T15:59:13.983 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph osd dump --format=json 2019-12-21T15:59:13.983 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: jq '.flags_set[]' 2019-12-21T15:59:13.983 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: grep -q '"noup"' 2019-12-21T15:59:13.983 INFO:tasks.workunit.client.0.smithi172.stdout:start osd.2 2019-12-21T15:59:14.003 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:59:14.000+0000 7f6b995f6ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T15:59:14.005 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:59:14.002+0000 7f6b995f6ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T15:59:14.006 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:59:14.003+0000 7f6b995f6ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T15:59:14.212 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: wait_for_osd up 2 2019-12-21T15:59:14.213 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:914: wait_for_osd: local state=up 2019-12-21T15:59:14.213 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local id=2 2019-12-21T15:59:14.213 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: status=1 2019-12-21T15:59:14.213 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i=0 )) 2019-12-21T15:59:14.215 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i < 300 )) 2019-12-21T15:59:14.215 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: echo 0 2019-12-21T15:59:14.216 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: ceph osd dump 2019-12-21T15:59:14.216 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: grep 'osd.2 up' 2019-12-21T15:59:14.216 INFO:tasks.workunit.client.0.smithi172.stdout:0 2019-12-21T15:59:14.442 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: sleep 1 2019-12-21T15:59:14.521 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:59:14.519+0000 7f6b995f6ec0 -1 Falling back to public interface 2019-12-21T15:59:15.248 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:59:15.245+0000 7f6b995f6ec0 -1 osd.2 0 log_to_monitors {default=true} 2019-12-21T15:59:15.444 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i++ )) 2019-12-21T15:59:15.444 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i < 300 )) 2019-12-21T15:59:15.444 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: echo 1 2019-12-21T15:59:15.444 INFO:tasks.workunit.client.0.smithi172.stdout:1 2019-12-21T15:59:15.445 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: ceph osd dump 2019-12-21T15:59:15.445 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: grep 'osd.2 up' 2019-12-21T15:59:15.701 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: sleep 1 2019-12-21T15:59:16.704 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i++ )) 2019-12-21T15:59:16.705 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i < 300 )) 2019-12-21T15:59:16.705 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: echo 2 2019-12-21T15:59:16.705 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: ceph osd dump 2019-12-21T15:59:16.717 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: grep 'osd.2 up' 2019-12-21T15:59:16.717 INFO:tasks.workunit.client.0.smithi172.stdout:2 2019-12-21T15:59:16.937 INFO:tasks.workunit.client.0.smithi172.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/128625,v1:127.0.0.1:6819/128625] [v2:127.0.0.1:6820/128625,v1:127.0.0.1:6821/128625] exists,up 9ebd74a3-4d1d-4717-b3f8-baa706f9765d 2019-12-21T15:59:16.937 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=0 2019-12-21T15:59:16.937 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: break 2019-12-21T15:59:16.938 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: return 0 2019-12-21T15:59:16.938 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:451: TEST_auto_repair_bluestore_failed: create_pool testpool 1 1 2019-12-21T15:59:16.938 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:539: create_pool: ceph osd pool create testpool 1 1 2019-12-21T15:59:17.352 INFO:tasks.workunit.client.0.smithi172.stderr:pool 'testpool' created 2019-12-21T15:59:17.357 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: sleep 1 2019-12-21T15:59:18.358 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:452: TEST_auto_repair_bluestore_failed: ceph osd pool set testpool size 2 2019-12-21T15:59:18.794 INFO:tasks.workunit.client.0.smithi172.stderr:set pool 1 size to 2 2019-12-21T15:59:18.804 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:453: TEST_auto_repair_bluestore_failed: wait_for_clean 2019-12-21T15:59:18.804 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1548: wait_for_clean: local cmd= 2019-12-21T15:59:18.804 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1549: wait_for_clean: local num_active_clean=-1 2019-12-21T15:59:18.805 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1550: wait_for_clean: local cur_active_clean 2019-12-21T15:59:18.805 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1551: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2019-12-21T15:59:18.805 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1551: wait_for_clean: get_timeout_delays 90 .1 2019-12-21T15:59:18.805 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1494: get_timeout_delays: shopt -q -o xtrace 2019-12-21T15:59:18.806 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1494: get_timeout_delays: echo true 2019-12-21T15:59:18.806 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1494: get_timeout_delays: local trace=true 2019-12-21T15:59:18.806 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1495: get_timeout_delays: true 2019-12-21T15:59:18.806 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1495: get_timeout_delays: shopt -u -o xtrace 2019-12-21T15:59:18.895 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1551: wait_for_clean: local -a delays 2019-12-21T15:59:18.895 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1552: wait_for_clean: local -i loop=0 2019-12-21T15:59:18.895 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1554: wait_for_clean: flush_pg_stats 2019-12-21T15:59:18.895 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1977: flush_pg_stats: local timeout=300 2019-12-21T15:59:18.896 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1979: flush_pg_stats: ceph osd ls 2019-12-21T15:59:19.125 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1979: flush_pg_stats: ids='0 2019-12-21T15:59:19.125 INFO:tasks.workunit.client.0.smithi172.stderr:1 2019-12-21T15:59:19.126 INFO:tasks.workunit.client.0.smithi172.stderr:2' 2019-12-21T15:59:19.126 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1980: flush_pg_stats: seqs= 2019-12-21T15:59:19.126 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: flush_pg_stats: for osd in $ids 2019-12-21T15:59:19.126 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2019-12-21T15:59:19.237 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: seq=21474836485 2019-12-21T15:59:19.237 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: flush_pg_stats: seqs=' 0-21474836485' 2019-12-21T15:59:19.237 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: flush_pg_stats: for osd in $ids 2019-12-21T15:59:19.237 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2019-12-21T15:59:19.347 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: seq=42949672963 2019-12-21T15:59:19.348 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: flush_pg_stats: seqs=' 0-21474836485 1-42949672963' 2019-12-21T15:59:19.348 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: flush_pg_stats: for osd in $ids 2019-12-21T15:59:19.348 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2019-12-21T15:59:19.459 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: seq=64424509443 2019-12-21T15:59:19.460 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: flush_pg_stats: seqs=' 0-21474836485 1-42949672963 2-64424509443' 2019-12-21T15:59:19.460 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: flush_pg_stats: for s in $seqs 2019-12-21T15:59:19.460 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: echo 0-21474836485 2019-12-21T15:59:19.460 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: cut -d - -f 1 2019-12-21T15:59:19.460 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: osd=0 2019-12-21T15:59:19.460 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: echo 0-21474836485 2019-12-21T15:59:19.461 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: cut -d - -f 2 2019-12-21T15:59:19.461 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: seq=21474836485 2019-12-21T15:59:19.461 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1989: flush_pg_stats: echo 'waiting osd.0 seq 21474836485' 2019-12-21T15:59:19.462 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 0 2019-12-21T15:59:19.462 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.0 seq 21474836485 2019-12-21T15:59:19.687 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 21474836484 -lt 21474836485 2019-12-21T15:59:19.687 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1991: flush_pg_stats: sleep 1 2019-12-21T15:59:20.688 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1992: flush_pg_stats: '[' 300 -eq 0 ']' 2019-12-21T15:59:20.689 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 0 2019-12-21T15:59:20.925 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 21474836485 -lt 21474836485 2019-12-21T15:59:20.926 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: flush_pg_stats: for s in $seqs 2019-12-21T15:59:20.926 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: echo 1-42949672963 2019-12-21T15:59:20.926 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: cut -d - -f 1 2019-12-21T15:59:20.927 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: osd=1 2019-12-21T15:59:20.928 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: echo 1-42949672963 2019-12-21T15:59:20.928 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: cut -d - -f 2 2019-12-21T15:59:20.929 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: seq=42949672963 2019-12-21T15:59:20.929 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1989: flush_pg_stats: echo 'waiting osd.1 seq 42949672963' 2019-12-21T15:59:20.929 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 1 2019-12-21T15:59:20.929 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.1 seq 42949672963 2019-12-21T15:59:21.155 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 42949672964 -lt 42949672963 2019-12-21T15:59:21.156 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: flush_pg_stats: for s in $seqs 2019-12-21T15:59:21.156 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: echo 2-64424509443 2019-12-21T15:59:21.156 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: cut -d - -f 1 2019-12-21T15:59:21.157 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: osd=2 2019-12-21T15:59:21.157 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: echo 2-64424509443 2019-12-21T15:59:21.157 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: cut -d - -f 2 2019-12-21T15:59:21.158 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: seq=64424509443 2019-12-21T15:59:21.159 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1989: flush_pg_stats: echo 'waiting osd.2 seq 64424509443' 2019-12-21T15:59:21.159 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 2 2019-12-21T15:59:21.159 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.2 seq 64424509443 2019-12-21T15:59:21.386 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 64424509443 -lt 64424509443 2019-12-21T15:59:21.386 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1555: wait_for_clean: get_num_pgs 2019-12-21T15:59:21.387 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: ceph --format json status 2019-12-21T15:59:21.387 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: jq .pgmap.num_pgs 2019-12-21T15:59:21.656 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1555: wait_for_clean: test 1 == 0 2019-12-21T15:59:21.656 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1559: wait_for_clean: true 2019-12-21T15:59:21.657 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1563: wait_for_clean: get_num_active_clean 2019-12-21T15:59:21.657 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: get_num_active_clean: local expression 2019-12-21T15:59:21.657 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2019-12-21T15:59:21.657 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("stale") | not)' 2019-12-21T15:59:21.657 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: ceph --format json pg dump pgs 2019-12-21T15:59:21.657 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2019-12-21T15:59:21.879 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1563: wait_for_clean: cur_active_clean=1 2019-12-21T15:59:21.879 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1564: wait_for_clean: get_num_pgs 2019-12-21T15:59:21.879 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: ceph --format json status 2019-12-21T15:59:21.879 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: jq .pgmap.num_pgs 2019-12-21T15:59:22.149 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1564: wait_for_clean: test 1 = 1 2019-12-21T15:59:22.149 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1564: wait_for_clean: break 2019-12-21T15:59:22.149 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: return 0 2019-12-21T15:59:22.150 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:456: TEST_auto_repair_bluestore_failed: local payload=ABCDEF 2019-12-21T15:59:22.150 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:457: TEST_auto_repair_bluestore_failed: echo ABCDEF 2019-12-21T15:59:22.150 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:458: TEST_auto_repair_bluestore_failed: seq 1 10 2019-12-21T15:59:22.150 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:458: TEST_auto_repair_bluestore_failed: for i in $(seq 1 10) 2019-12-21T15:59:22.151 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:460: TEST_auto_repair_bluestore_failed: rados --pool testpool put obj1 td/osd-scrub-repair/ORIGINAL 2019-12-21T15:59:22.191 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:458: TEST_auto_repair_bluestore_failed: for i in $(seq 1 10) 2019-12-21T15:59:22.191 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:460: TEST_auto_repair_bluestore_failed: rados --pool testpool put obj2 td/osd-scrub-repair/ORIGINAL 2019-12-21T15:59:22.272 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:458: TEST_auto_repair_bluestore_failed: for i in $(seq 1 10) 2019-12-21T15:59:22.273 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:460: TEST_auto_repair_bluestore_failed: rados --pool testpool put obj3 td/osd-scrub-repair/ORIGINAL 2019-12-21T15:59:22.311 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:458: TEST_auto_repair_bluestore_failed: for i in $(seq 1 10) 2019-12-21T15:59:22.311 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:460: TEST_auto_repair_bluestore_failed: rados --pool testpool put obj4 td/osd-scrub-repair/ORIGINAL 2019-12-21T15:59:22.349 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:458: TEST_auto_repair_bluestore_failed: for i in $(seq 1 10) 2019-12-21T15:59:22.350 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:460: TEST_auto_repair_bluestore_failed: rados --pool testpool put obj5 td/osd-scrub-repair/ORIGINAL 2019-12-21T15:59:22.386 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:458: TEST_auto_repair_bluestore_failed: for i in $(seq 1 10) 2019-12-21T15:59:22.386 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:460: TEST_auto_repair_bluestore_failed: rados --pool testpool put obj6 td/osd-scrub-repair/ORIGINAL 2019-12-21T15:59:22.424 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:458: TEST_auto_repair_bluestore_failed: for i in $(seq 1 10) 2019-12-21T15:59:22.425 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:460: TEST_auto_repair_bluestore_failed: rados --pool testpool put obj7 td/osd-scrub-repair/ORIGINAL 2019-12-21T15:59:22.458 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:458: TEST_auto_repair_bluestore_failed: for i in $(seq 1 10) 2019-12-21T15:59:22.458 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:460: TEST_auto_repair_bluestore_failed: rados --pool testpool put obj8 td/osd-scrub-repair/ORIGINAL 2019-12-21T15:59:22.521 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:458: TEST_auto_repair_bluestore_failed: for i in $(seq 1 10) 2019-12-21T15:59:22.521 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:460: TEST_auto_repair_bluestore_failed: rados --pool testpool put obj9 td/osd-scrub-repair/ORIGINAL 2019-12-21T15:59:22.553 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:458: TEST_auto_repair_bluestore_failed: for i in $(seq 1 10) 2019-12-21T15:59:22.553 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:460: TEST_auto_repair_bluestore_failed: rados --pool testpool put obj10 td/osd-scrub-repair/ORIGINAL 2019-12-21T15:59:22.592 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:465: TEST_auto_repair_bluestore_failed: get_not_primary testpool SOMETHING 2019-12-21T15:59:22.592 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1165: get_not_primary: local poolname=testpool 2019-12-21T15:59:22.592 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1166: get_not_primary: local objectname=SOMETHING 2019-12-21T15:59:22.593 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: get_not_primary: get_primary testpool SOMETHING 2019-12-21T15:59:22.593 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_primary: local poolname=testpool 2019-12-21T15:59:22.593 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1133: get_primary: local objectname=SOMETHING 2019-12-21T15:59:22.593 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1135: get_primary: ceph --format json osd map testpool SOMETHING 2019-12-21T15:59:22.593 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1136: get_primary: jq .acting_primary 2019-12-21T15:59:22.822 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: get_not_primary: local primary=1 2019-12-21T15:59:22.822 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1169: get_not_primary: ceph --format json osd map testpool SOMETHING 2019-12-21T15:59:22.822 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1170: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2019-12-21T15:59:23.052 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:465: TEST_auto_repair_bluestore_failed: objectstore_tool td/osd-scrub-repair 0 obj1 remove 2019-12-21T15:59:23.052 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: objectstore_tool: local dir=td/osd-scrub-repair 2019-12-21T15:59:23.052 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: objectstore_tool: shift 2019-12-21T15:59:23.052 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1235: objectstore_tool: local id=0 2019-12-21T15:59:23.053 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1236: objectstore_tool: shift 2019-12-21T15:59:23.053 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1238: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 obj1 remove 2019-12-21T15:59:23.053 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1205: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2019-12-21T15:59:23.053 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nowait: shift 2019-12-21T15:59:23.053 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1207: _objectstore_tool_nowait: local id=0 2019-12-21T15:59:23.053 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1208: _objectstore_tool_nowait: shift 2019-12-21T15:59:23.053 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1210: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2019-12-21T15:59:23.054 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2019-12-21T15:59:23.054 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2019-12-21T15:59:23.054 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2019-12-21T15:59:23.054 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2019-12-21T15:59:23.054 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2019-12-21T15:59:23.157 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2019-12-21T15:59:23.158 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1212: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 obj1 remove 2019-12-21T15:59:23.158 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1193: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2019-12-21T15:59:23.158 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1194: _objectstore_tool_nodown: shift 2019-12-21T15:59:23.159 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1195: _objectstore_tool_nodown: local id=0 2019-12-21T15:59:23.159 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: _objectstore_tool_nodown: shift 2019-12-21T15:59:23.159 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2019-12-21T15:59:23.159 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj1 remove 2019-12-21T15:59:23.782 INFO:tasks.workunit.client.0.smithi172.stdout:remove #1:ff7b1f36:::obj1:head# 2019-12-21T15:59:24.302 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1213: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2019-12-21T15:59:24.303 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:842: activate_osd: local dir=td/osd-scrub-repair 2019-12-21T15:59:24.303 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: shift 2019-12-21T15:59:24.303 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: local id=0 2019-12-21T15:59:24.303 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: shift 2019-12-21T15:59:24.303 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local osd_data=td/osd-scrub-repair/0 2019-12-21T15:59:24.303 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local 'ceph_args=--fsid=973e3fc1-e63e-4bde-9db6-84c68fd5e93e --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2019-12-21T15:59:24.304 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2019-12-21T15:59:24.304 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-journal-size=100' 2019-12-21T15:59:24.304 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2019-12-21T15:59:24.304 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2019-12-21T15:59:24.304 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2019-12-21T15:59:24.304 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --chdir=' 2019-12-21T15:59:24.304 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+= 2019-12-21T15:59:24.305 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2019-12-21T15:59:24.305 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: get_asok_path 2019-12-21T15:59:24.305 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2019-12-21T15:59:24.305 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2019-12-21T15:59:24.305 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2019-12-21T15:59:24.305 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:59:24.305 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.119575 2019-12-21T15:59:24.306 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.119575/$cluster-$name.asok' 2019-12-21T15:59:24.306 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.119575/$cluster-$name.asok' 2019-12-21T15:59:24.306 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --debug-osd=20' 2019-12-21T15:59:24.306 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2019-12-21T15:59:24.306 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2019-12-21T15:59:24.306 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2019-12-21T15:59:24.308 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2019-12-21T15:59:24.308 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2019-12-21T15:59:24.308 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' ' 2019-12-21T15:59:24.309 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2019-12-21T15:59:24.309 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: mkdir -p td/osd-scrub-repair/0 2019-12-21T15:59:24.309 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: echo start osd.0 2019-12-21T15:59:24.309 INFO:tasks.workunit.client.0.smithi172.stderr:start osd.0 2019-12-21T15:59:24.309 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph-osd -i 0 --fsid=973e3fc1-e63e-4bde-9db6-84c68fd5e93e --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.119575/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2019-12-21T15:59:24.309 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: cat td/osd-scrub-repair/0/whoami 2019-12-21T15:59:24.309 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: '[' 0 = 0 ']' 2019-12-21T15:59:24.310 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph osd dump --format=json 2019-12-21T15:59:24.310 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: jq '.flags_set[]' 2019-12-21T15:59:24.310 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: grep -q '"noup"' 2019-12-21T15:59:24.324 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:59:24.322+0000 7fd8223d5ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T15:59:24.326 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:59:24.324+0000 7fd8223d5ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T15:59:24.329 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:59:24.327+0000 7fd8223d5ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T15:59:24.544 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: wait_for_osd up 0 2019-12-21T15:59:24.544 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:914: wait_for_osd: local state=up 2019-12-21T15:59:24.544 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local id=0 2019-12-21T15:59:24.544 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: status=1 2019-12-21T15:59:24.544 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i=0 )) 2019-12-21T15:59:24.545 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i < 300 )) 2019-12-21T15:59:24.545 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: echo 0 2019-12-21T15:59:24.545 INFO:tasks.workunit.client.0.smithi172.stderr:0 2019-12-21T15:59:24.545 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: ceph osd dump 2019-12-21T15:59:24.545 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: grep 'osd.0 up' 2019-12-21T15:59:24.786 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: sleep 1 2019-12-21T15:59:24.848 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:59:24.846+0000 7fd8223d5ec0 -1 Falling back to public interface 2019-12-21T15:59:25.609 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:59:25.607+0000 7fd8223d5ec0 -1 osd.0 21 log_to_monitors {default=true} 2019-12-21T15:59:25.787 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i++ )) 2019-12-21T15:59:25.787 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i < 300 )) 2019-12-21T15:59:25.787 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: echo 1 2019-12-21T15:59:25.788 INFO:tasks.workunit.client.0.smithi172.stderr:1 2019-12-21T15:59:25.788 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: grep 'osd.0 up' 2019-12-21T15:59:25.788 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: ceph osd dump 2019-12-21T15:59:26.062 INFO:tasks.workunit.client.0.smithi172.stderr:osd.0 up in weight 1 up_from 24 up_thru 0 down_at 22 last_clean_interval [5,21) [v2:127.0.0.1:6802/129641,v1:127.0.0.1:6803/129641] [v2:127.0.0.1:6804/129641,v1:127.0.0.1:6805/129641] exists,up 4236b440-d79c-4b13-9ee0-6323732a2f6e 2019-12-21T15:59:26.062 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=0 2019-12-21T15:59:26.063 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: break 2019-12-21T15:59:26.063 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: return 0 2019-12-21T15:59:26.063 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1239: objectstore_tool: wait_for_clean 2019-12-21T15:59:26.063 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1548: wait_for_clean: local cmd= 2019-12-21T15:59:26.063 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1549: wait_for_clean: local num_active_clean=-1 2019-12-21T15:59:26.063 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1550: wait_for_clean: local cur_active_clean 2019-12-21T15:59:26.064 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1551: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2019-12-21T15:59:26.064 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1551: wait_for_clean: get_timeout_delays 90 .1 2019-12-21T15:59:26.064 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1494: get_timeout_delays: shopt -q -o xtrace 2019-12-21T15:59:26.064 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1494: get_timeout_delays: echo true 2019-12-21T15:59:26.064 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1494: get_timeout_delays: local trace=true 2019-12-21T15:59:26.064 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1495: get_timeout_delays: true 2019-12-21T15:59:26.064 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1495: get_timeout_delays: shopt -u -o xtrace 2019-12-21T15:59:26.155 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1551: wait_for_clean: local -a delays 2019-12-21T15:59:26.155 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1552: wait_for_clean: local -i loop=0 2019-12-21T15:59:26.156 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1554: wait_for_clean: flush_pg_stats 2019-12-21T15:59:26.157 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1977: flush_pg_stats: local timeout=300 2019-12-21T15:59:26.157 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1979: flush_pg_stats: ceph osd ls 2019-12-21T15:59:26.395 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1979: flush_pg_stats: ids='0 2019-12-21T15:59:26.395 INFO:tasks.workunit.client.0.smithi172.stderr:1 2019-12-21T15:59:26.395 INFO:tasks.workunit.client.0.smithi172.stderr:2' 2019-12-21T15:59:26.395 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1980: flush_pg_stats: seqs= 2019-12-21T15:59:26.396 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: flush_pg_stats: for osd in $ids 2019-12-21T15:59:26.396 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2019-12-21T15:59:26.515 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: seq=103079215106 2019-12-21T15:59:26.515 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: flush_pg_stats: seqs=' 0-103079215106' 2019-12-21T15:59:26.515 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: flush_pg_stats: for osd in $ids 2019-12-21T15:59:26.516 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2019-12-21T15:59:26.626 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: seq=42949672966 2019-12-21T15:59:26.626 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: flush_pg_stats: seqs=' 0-103079215106 1-42949672966' 2019-12-21T15:59:26.626 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: flush_pg_stats: for osd in $ids 2019-12-21T15:59:26.626 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2019-12-21T15:59:26.738 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: seq=64424509446 2019-12-21T15:59:26.738 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: flush_pg_stats: seqs=' 0-103079215106 1-42949672966 2-64424509446' 2019-12-21T15:59:26.738 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: flush_pg_stats: for s in $seqs 2019-12-21T15:59:26.739 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: echo 0-103079215106 2019-12-21T15:59:26.739 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: cut -d - -f 1 2019-12-21T15:59:26.739 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: osd=0 2019-12-21T15:59:26.740 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: echo 0-103079215106 2019-12-21T15:59:26.740 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: cut -d - -f 2 2019-12-21T15:59:26.740 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: seq=103079215106 2019-12-21T15:59:26.740 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1989: flush_pg_stats: echo 'waiting osd.0 seq 103079215106' 2019-12-21T15:59:26.741 INFO:tasks.workunit.client.0.smithi172.stderr:waiting osd.0 seq 103079215106 2019-12-21T15:59:26.741 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 0 2019-12-21T15:59:26.968 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 1 -lt 103079215106 2019-12-21T15:59:26.968 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1991: flush_pg_stats: sleep 1 2019-12-21T15:59:27.969 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1992: flush_pg_stats: '[' 300 -eq 0 ']' 2019-12-21T15:59:27.970 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 0 2019-12-21T15:59:28.207 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 1 -lt 103079215106 2019-12-21T15:59:28.207 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1991: flush_pg_stats: sleep 1 2019-12-21T15:59:29.208 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1992: flush_pg_stats: '[' 299 -eq 0 ']' 2019-12-21T15:59:29.208 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 0 2019-12-21T15:59:29.445 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 103079215106 -lt 103079215106 2019-12-21T15:59:29.446 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: flush_pg_stats: for s in $seqs 2019-12-21T15:59:29.446 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: echo 1-42949672966 2019-12-21T15:59:29.446 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: cut -d - -f 1 2019-12-21T15:59:29.447 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: osd=1 2019-12-21T15:59:29.447 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: echo 1-42949672966 2019-12-21T15:59:29.447 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: cut -d - -f 2 2019-12-21T15:59:29.449 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: seq=42949672966 2019-12-21T15:59:29.450 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1989: flush_pg_stats: echo 'waiting osd.1 seq 42949672966' 2019-12-21T15:59:29.450 INFO:tasks.workunit.client.0.smithi172.stderr:waiting osd.1 seq 42949672966 2019-12-21T15:59:29.450 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 1 2019-12-21T15:59:29.677 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 42949672966 -lt 42949672966 2019-12-21T15:59:29.678 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: flush_pg_stats: for s in $seqs 2019-12-21T15:59:29.678 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: echo 2-64424509446 2019-12-21T15:59:29.678 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: cut -d - -f 1 2019-12-21T15:59:29.679 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: osd=2 2019-12-21T15:59:29.679 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: echo 2-64424509446 2019-12-21T15:59:29.679 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: cut -d - -f 2 2019-12-21T15:59:29.681 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: seq=64424509446 2019-12-21T15:59:29.681 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1989: flush_pg_stats: echo 'waiting osd.2 seq 64424509446' 2019-12-21T15:59:29.681 INFO:tasks.workunit.client.0.smithi172.stderr:waiting osd.2 seq 64424509446 2019-12-21T15:59:29.681 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 2 2019-12-21T15:59:29.907 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 64424509446 -lt 64424509446 2019-12-21T15:59:29.908 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1555: wait_for_clean: get_num_pgs 2019-12-21T15:59:29.908 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: ceph --format json status 2019-12-21T15:59:29.908 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: jq .pgmap.num_pgs 2019-12-21T15:59:30.176 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1555: wait_for_clean: test 1 == 0 2019-12-21T15:59:30.176 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1559: wait_for_clean: true 2019-12-21T15:59:30.177 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1563: wait_for_clean: get_num_active_clean 2019-12-21T15:59:30.177 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: get_num_active_clean: local expression 2019-12-21T15:59:30.177 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2019-12-21T15:59:30.177 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("stale") | not)' 2019-12-21T15:59:30.177 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: ceph --format json pg dump pgs 2019-12-21T15:59:30.177 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2019-12-21T15:59:30.399 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1563: wait_for_clean: cur_active_clean=1 2019-12-21T15:59:30.399 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1564: wait_for_clean: get_num_pgs 2019-12-21T15:59:30.400 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: ceph --format json status 2019-12-21T15:59:30.400 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: jq .pgmap.num_pgs 2019-12-21T15:59:30.687 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1564: wait_for_clean: test 1 = 1 2019-12-21T15:59:30.688 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1564: wait_for_clean: break 2019-12-21T15:59:30.688 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: return 0 2019-12-21T15:59:30.688 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:467: TEST_auto_repair_bluestore_failed: get_not_primary testpool SOMETHING 2019-12-21T15:59:30.688 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1165: get_not_primary: local poolname=testpool 2019-12-21T15:59:30.688 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1166: get_not_primary: local objectname=SOMETHING 2019-12-21T15:59:30.689 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: get_not_primary: get_primary testpool SOMETHING 2019-12-21T15:59:30.689 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_primary: local poolname=testpool 2019-12-21T15:59:30.689 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1133: get_primary: local objectname=SOMETHING 2019-12-21T15:59:30.689 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1135: get_primary: ceph --format json osd map testpool SOMETHING 2019-12-21T15:59:30.690 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1136: get_primary: jq .acting_primary 2019-12-21T15:59:30.919 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: get_not_primary: local primary=1 2019-12-21T15:59:30.920 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1169: get_not_primary: ceph --format json osd map testpool SOMETHING 2019-12-21T15:59:30.920 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1170: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2019-12-21T15:59:31.149 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:467: TEST_auto_repair_bluestore_failed: objectstore_tool td/osd-scrub-repair 0 obj2 remove 2019-12-21T15:59:31.149 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: objectstore_tool: local dir=td/osd-scrub-repair 2019-12-21T15:59:31.149 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: objectstore_tool: shift 2019-12-21T15:59:31.149 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1235: objectstore_tool: local id=0 2019-12-21T15:59:31.150 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1236: objectstore_tool: shift 2019-12-21T15:59:31.150 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1238: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 obj2 remove 2019-12-21T15:59:31.150 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1205: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2019-12-21T15:59:31.150 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nowait: shift 2019-12-21T15:59:31.150 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1207: _objectstore_tool_nowait: local id=0 2019-12-21T15:59:31.150 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1208: _objectstore_tool_nowait: shift 2019-12-21T15:59:31.150 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1210: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2019-12-21T15:59:31.151 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2019-12-21T15:59:31.151 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2019-12-21T15:59:31.151 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2019-12-21T15:59:31.151 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2019-12-21T15:59:31.151 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2019-12-21T15:59:31.254 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2019-12-21T15:59:31.254 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1212: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 obj2 remove 2019-12-21T15:59:31.254 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1193: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2019-12-21T15:59:31.254 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1194: _objectstore_tool_nodown: shift 2019-12-21T15:59:31.255 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1195: _objectstore_tool_nodown: local id=0 2019-12-21T15:59:31.255 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: _objectstore_tool_nodown: shift 2019-12-21T15:59:31.255 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2019-12-21T15:59:31.255 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj2 remove 2019-12-21T15:59:31.845 INFO:tasks.workunit.client.0.smithi172.stdout:remove #1:104778fc:::obj2:head# 2019-12-21T15:59:32.363 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1213: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2019-12-21T15:59:32.364 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:842: activate_osd: local dir=td/osd-scrub-repair 2019-12-21T15:59:32.364 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: shift 2019-12-21T15:59:32.364 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: local id=0 2019-12-21T15:59:32.364 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: shift 2019-12-21T15:59:32.364 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local osd_data=td/osd-scrub-repair/0 2019-12-21T15:59:32.364 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local 'ceph_args=--fsid=973e3fc1-e63e-4bde-9db6-84c68fd5e93e --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2019-12-21T15:59:32.365 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2019-12-21T15:59:32.365 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-journal-size=100' 2019-12-21T15:59:32.365 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2019-12-21T15:59:32.365 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2019-12-21T15:59:32.365 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2019-12-21T15:59:32.365 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --chdir=' 2019-12-21T15:59:32.365 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+= 2019-12-21T15:59:32.366 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2019-12-21T15:59:32.366 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: get_asok_path 2019-12-21T15:59:32.366 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2019-12-21T15:59:32.366 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2019-12-21T15:59:32.366 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2019-12-21T15:59:32.366 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:59:32.366 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.119575 2019-12-21T15:59:32.367 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.119575/$cluster-$name.asok' 2019-12-21T15:59:32.367 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.119575/$cluster-$name.asok' 2019-12-21T15:59:32.367 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --debug-osd=20' 2019-12-21T15:59:32.367 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2019-12-21T15:59:32.367 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2019-12-21T15:59:32.367 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2019-12-21T15:59:32.367 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2019-12-21T15:59:32.368 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2019-12-21T15:59:32.368 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' ' 2019-12-21T15:59:32.368 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2019-12-21T15:59:32.368 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: mkdir -p td/osd-scrub-repair/0 2019-12-21T15:59:32.368 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: echo start osd.0 2019-12-21T15:59:32.368 INFO:tasks.workunit.client.0.smithi172.stderr:start osd.0 2019-12-21T15:59:32.369 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph-osd -i 0 --fsid=973e3fc1-e63e-4bde-9db6-84c68fd5e93e --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.119575/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2019-12-21T15:59:32.369 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: cat td/osd-scrub-repair/0/whoami 2019-12-21T15:59:32.370 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: '[' 0 = 0 ']' 2019-12-21T15:59:32.370 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph osd dump --format=json 2019-12-21T15:59:32.370 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: jq '.flags_set[]' 2019-12-21T15:59:32.370 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: grep -q '"noup"' 2019-12-21T15:59:32.387 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:59:32.385+0000 7f6f53509ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T15:59:32.388 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:59:32.387+0000 7f6f53509ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T15:59:32.390 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:59:32.388+0000 7f6f53509ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T15:59:32.603 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: wait_for_osd up 0 2019-12-21T15:59:32.603 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:914: wait_for_osd: local state=up 2019-12-21T15:59:32.603 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local id=0 2019-12-21T15:59:32.603 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: status=1 2019-12-21T15:59:32.603 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i=0 )) 2019-12-21T15:59:32.604 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i < 300 )) 2019-12-21T15:59:32.604 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: echo 0 2019-12-21T15:59:32.604 INFO:tasks.workunit.client.0.smithi172.stderr:0 2019-12-21T15:59:32.604 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: ceph osd dump 2019-12-21T15:59:32.604 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: grep 'osd.0 up' 2019-12-21T15:59:32.831 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: sleep 1 2019-12-21T15:59:32.907 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:59:32.905+0000 7f6f53509ec0 -1 Falling back to public interface 2019-12-21T15:59:33.635 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:59:33.633+0000 7f6f53509ec0 -1 osd.0 25 log_to_monitors {default=true} 2019-12-21T15:59:33.832 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i++ )) 2019-12-21T15:59:33.833 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i < 300 )) 2019-12-21T15:59:33.833 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: echo 1 2019-12-21T15:59:33.833 INFO:tasks.workunit.client.0.smithi172.stderr:1 2019-12-21T15:59:33.833 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: ceph osd dump 2019-12-21T15:59:33.833 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: grep 'osd.0 up' 2019-12-21T15:59:34.060 INFO:tasks.workunit.client.0.smithi172.stderr:osd.0 up in weight 1 up_from 28 up_thru 0 down_at 26 last_clean_interval [24,25) [v2:127.0.0.1:6802/130382,v1:127.0.0.1:6803/130382] [v2:127.0.0.1:6804/130382,v1:127.0.0.1:6805/130382] exists,up 4236b440-d79c-4b13-9ee0-6323732a2f6e 2019-12-21T15:59:34.061 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=0 2019-12-21T15:59:34.061 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: break 2019-12-21T15:59:34.061 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: return 0 2019-12-21T15:59:34.061 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1239: objectstore_tool: wait_for_clean 2019-12-21T15:59:34.061 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1548: wait_for_clean: local cmd= 2019-12-21T15:59:34.061 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1549: wait_for_clean: local num_active_clean=-1 2019-12-21T15:59:34.062 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1550: wait_for_clean: local cur_active_clean 2019-12-21T15:59:34.062 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1551: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2019-12-21T15:59:34.062 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1551: wait_for_clean: get_timeout_delays 90 .1 2019-12-21T15:59:34.062 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1494: get_timeout_delays: shopt -q -o xtrace 2019-12-21T15:59:34.062 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1494: get_timeout_delays: echo true 2019-12-21T15:59:34.062 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1494: get_timeout_delays: local trace=true 2019-12-21T15:59:34.062 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1495: get_timeout_delays: true 2019-12-21T15:59:34.063 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1495: get_timeout_delays: shopt -u -o xtrace 2019-12-21T15:59:34.153 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1551: wait_for_clean: local -a delays 2019-12-21T15:59:34.153 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1552: wait_for_clean: local -i loop=0 2019-12-21T15:59:34.154 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1554: wait_for_clean: flush_pg_stats 2019-12-21T15:59:34.154 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1977: flush_pg_stats: local timeout=300 2019-12-21T15:59:34.154 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1979: flush_pg_stats: ceph osd ls 2019-12-21T15:59:34.381 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1979: flush_pg_stats: ids='0 2019-12-21T15:59:34.381 INFO:tasks.workunit.client.0.smithi172.stderr:1 2019-12-21T15:59:34.381 INFO:tasks.workunit.client.0.smithi172.stderr:2' 2019-12-21T15:59:34.381 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1980: flush_pg_stats: seqs= 2019-12-21T15:59:34.381 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: flush_pg_stats: for osd in $ids 2019-12-21T15:59:34.381 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2019-12-21T15:59:34.508 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: seq=120259084290 2019-12-21T15:59:34.508 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: flush_pg_stats: seqs=' 0-120259084290' 2019-12-21T15:59:34.508 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: flush_pg_stats: for osd in $ids 2019-12-21T15:59:34.508 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2019-12-21T15:59:34.618 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: seq=42949672968 2019-12-21T15:59:34.618 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: flush_pg_stats: seqs=' 0-120259084290 1-42949672968' 2019-12-21T15:59:34.618 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: flush_pg_stats: for osd in $ids 2019-12-21T15:59:34.618 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2019-12-21T15:59:34.729 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: seq=64424509448 2019-12-21T15:59:34.729 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: flush_pg_stats: seqs=' 0-120259084290 1-42949672968 2-64424509448' 2019-12-21T15:59:34.729 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: flush_pg_stats: for s in $seqs 2019-12-21T15:59:34.730 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: echo 0-120259084290 2019-12-21T15:59:34.730 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: cut -d - -f 1 2019-12-21T15:59:34.730 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: osd=0 2019-12-21T15:59:34.731 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: echo 0-120259084290 2019-12-21T15:59:34.731 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: cut -d - -f 2 2019-12-21T15:59:34.732 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: seq=120259084290 2019-12-21T15:59:34.732 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1989: flush_pg_stats: echo 'waiting osd.0 seq 120259084290' 2019-12-21T15:59:34.732 INFO:tasks.workunit.client.0.smithi172.stderr:waiting osd.0 seq 120259084290 2019-12-21T15:59:34.732 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 0 2019-12-21T15:59:34.962 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 1 -lt 120259084290 2019-12-21T15:59:34.962 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1991: flush_pg_stats: sleep 1 2019-12-21T15:59:35.963 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1992: flush_pg_stats: '[' 300 -eq 0 ']' 2019-12-21T15:59:35.963 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 0 2019-12-21T15:59:36.197 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 1 -lt 120259084290 2019-12-21T15:59:36.197 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1991: flush_pg_stats: sleep 1 2019-12-21T15:59:37.198 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1992: flush_pg_stats: '[' 299 -eq 0 ']' 2019-12-21T15:59:37.198 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 0 2019-12-21T15:59:37.427 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 120259084290 -lt 120259084290 2019-12-21T15:59:37.427 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: flush_pg_stats: for s in $seqs 2019-12-21T15:59:37.428 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: echo 1-42949672968 2019-12-21T15:59:37.428 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: cut -d - -f 1 2019-12-21T15:59:37.428 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: osd=1 2019-12-21T15:59:37.429 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: echo 1-42949672968 2019-12-21T15:59:37.429 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: cut -d - -f 2 2019-12-21T15:59:37.429 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: seq=42949672968 2019-12-21T15:59:37.430 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1989: flush_pg_stats: echo 'waiting osd.1 seq 42949672968' 2019-12-21T15:59:37.430 INFO:tasks.workunit.client.0.smithi172.stderr:waiting osd.1 seq 42949672968 2019-12-21T15:59:37.430 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 1 2019-12-21T15:59:37.658 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 42949672969 -lt 42949672968 2019-12-21T15:59:37.658 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: flush_pg_stats: for s in $seqs 2019-12-21T15:59:37.658 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: echo 2-64424509448 2019-12-21T15:59:37.658 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: cut -d - -f 1 2019-12-21T15:59:37.659 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: osd=2 2019-12-21T15:59:37.659 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: echo 2-64424509448 2019-12-21T15:59:37.660 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: cut -d - -f 2 2019-12-21T15:59:37.660 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: seq=64424509448 2019-12-21T15:59:37.661 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1989: flush_pg_stats: echo 'waiting osd.2 seq 64424509448' 2019-12-21T15:59:37.661 INFO:tasks.workunit.client.0.smithi172.stderr:waiting osd.2 seq 64424509448 2019-12-21T15:59:37.661 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 2 2019-12-21T15:59:37.888 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 64424509448 -lt 64424509448 2019-12-21T15:59:37.889 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1555: wait_for_clean: get_num_pgs 2019-12-21T15:59:37.889 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: ceph --format json status 2019-12-21T15:59:37.889 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: jq .pgmap.num_pgs 2019-12-21T15:59:38.156 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1555: wait_for_clean: test 1 == 0 2019-12-21T15:59:38.156 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1559: wait_for_clean: true 2019-12-21T15:59:38.156 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1563: wait_for_clean: get_num_active_clean 2019-12-21T15:59:38.156 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: get_num_active_clean: local expression 2019-12-21T15:59:38.156 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2019-12-21T15:59:38.157 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("stale") | not)' 2019-12-21T15:59:38.157 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: ceph --format json pg dump pgs 2019-12-21T15:59:38.157 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2019-12-21T15:59:38.378 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1563: wait_for_clean: cur_active_clean=1 2019-12-21T15:59:38.378 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1564: wait_for_clean: get_num_pgs 2019-12-21T15:59:38.379 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: ceph --format json status 2019-12-21T15:59:38.379 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: jq .pgmap.num_pgs 2019-12-21T15:59:38.742 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1564: wait_for_clean: test 1 = 1 2019-12-21T15:59:38.742 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1564: wait_for_clean: break 2019-12-21T15:59:38.742 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: return 0 2019-12-21T15:59:38.742 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:468: TEST_auto_repair_bluestore_failed: get_primary testpool SOMETHING 2019-12-21T15:59:38.742 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_primary: local poolname=testpool 2019-12-21T15:59:38.742 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1133: get_primary: local objectname=SOMETHING 2019-12-21T15:59:38.743 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1135: get_primary: ceph --format json osd map testpool SOMETHING 2019-12-21T15:59:38.743 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1136: get_primary: jq .acting_primary 2019-12-21T15:59:38.974 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:468: TEST_auto_repair_bluestore_failed: objectstore_tool td/osd-scrub-repair 1 obj2 rm-attr _ 2019-12-21T15:59:38.974 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: objectstore_tool: local dir=td/osd-scrub-repair 2019-12-21T15:59:38.974 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: objectstore_tool: shift 2019-12-21T15:59:38.975 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1235: objectstore_tool: local id=1 2019-12-21T15:59:38.975 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1236: objectstore_tool: shift 2019-12-21T15:59:38.975 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1238: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 obj2 rm-attr _ 2019-12-21T15:59:38.975 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1205: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2019-12-21T15:59:38.975 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nowait: shift 2019-12-21T15:59:38.975 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1207: _objectstore_tool_nowait: local id=1 2019-12-21T15:59:38.975 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1208: _objectstore_tool_nowait: shift 2019-12-21T15:59:38.976 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1210: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2019-12-21T15:59:38.976 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2019-12-21T15:59:38.976 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2019-12-21T15:59:38.976 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2019-12-21T15:59:38.976 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2019-12-21T15:59:38.976 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2019-12-21T15:59:39.079 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2019-12-21T15:59:39.079 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1212: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 obj2 rm-attr _ 2019-12-21T15:59:39.080 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1193: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2019-12-21T15:59:39.080 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1194: _objectstore_tool_nodown: shift 2019-12-21T15:59:39.080 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1195: _objectstore_tool_nodown: local id=1 2019-12-21T15:59:39.080 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: _objectstore_tool_nodown: shift 2019-12-21T15:59:39.080 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2019-12-21T15:59:39.080 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj2 rm-attr _ 2019-12-21T15:59:40.308 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1213: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2019-12-21T15:59:40.308 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:842: activate_osd: local dir=td/osd-scrub-repair 2019-12-21T15:59:40.308 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: shift 2019-12-21T15:59:40.308 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: local id=1 2019-12-21T15:59:40.308 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: shift 2019-12-21T15:59:40.308 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local osd_data=td/osd-scrub-repair/1 2019-12-21T15:59:40.309 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local 'ceph_args=--fsid=973e3fc1-e63e-4bde-9db6-84c68fd5e93e --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2019-12-21T15:59:40.309 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2019-12-21T15:59:40.309 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-journal-size=100' 2019-12-21T15:59:40.309 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2019-12-21T15:59:40.309 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2019-12-21T15:59:40.309 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2019-12-21T15:59:40.309 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --chdir=' 2019-12-21T15:59:40.310 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+= 2019-12-21T15:59:40.310 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2019-12-21T15:59:40.310 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: get_asok_path 2019-12-21T15:59:40.310 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2019-12-21T15:59:40.310 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2019-12-21T15:59:40.310 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2019-12-21T15:59:40.311 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T15:59:40.311 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.119575 2019-12-21T15:59:40.311 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.119575/$cluster-$name.asok' 2019-12-21T15:59:40.311 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.119575/$cluster-$name.asok' 2019-12-21T15:59:40.311 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --debug-osd=20' 2019-12-21T15:59:40.311 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2019-12-21T15:59:40.311 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2019-12-21T15:59:40.312 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2019-12-21T15:59:40.312 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2019-12-21T15:59:40.312 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2019-12-21T15:59:40.312 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' ' 2019-12-21T15:59:40.312 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2019-12-21T15:59:40.312 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: mkdir -p td/osd-scrub-repair/1 2019-12-21T15:59:40.313 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: echo start osd.1 2019-12-21T15:59:40.313 INFO:tasks.workunit.client.0.smithi172.stderr:start osd.1 2019-12-21T15:59:40.314 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph-osd -i 1 --fsid=973e3fc1-e63e-4bde-9db6-84c68fd5e93e --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.119575/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2019-12-21T15:59:40.314 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: cat td/osd-scrub-repair/1/whoami 2019-12-21T15:59:40.314 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: '[' 1 = 1 ']' 2019-12-21T15:59:40.314 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph osd dump --format=json 2019-12-21T15:59:40.314 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: jq '.flags_set[]' 2019-12-21T15:59:40.314 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: grep -q '"noup"' 2019-12-21T15:59:40.332 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:59:40.331+0000 7fea1180bec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T15:59:40.334 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:59:40.333+0000 7fea1180bec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T15:59:40.336 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:59:40.334+0000 7fea1180bec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T15:59:40.549 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: wait_for_osd up 1 2019-12-21T15:59:40.549 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:914: wait_for_osd: local state=up 2019-12-21T15:59:40.550 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local id=1 2019-12-21T15:59:40.550 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: status=1 2019-12-21T15:59:40.550 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i=0 )) 2019-12-21T15:59:40.550 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i < 300 )) 2019-12-21T15:59:40.550 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: echo 0 2019-12-21T15:59:40.550 INFO:tasks.workunit.client.0.smithi172.stderr:0 2019-12-21T15:59:40.550 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: ceph osd dump 2019-12-21T15:59:40.551 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: grep 'osd.1 up' 2019-12-21T15:59:40.796 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: sleep 1 2019-12-21T15:59:40.857 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:59:40.856+0000 7fea1180bec0 -1 Falling back to public interface 2019-12-21T15:59:41.642 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T15:59:41.639+0000 7fea1180bec0 -1 osd.1 29 log_to_monitors {default=true} 2019-12-21T15:59:41.797 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i++ )) 2019-12-21T15:59:41.798 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i < 300 )) 2019-12-21T15:59:41.798 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: echo 1 2019-12-21T15:59:41.798 INFO:tasks.workunit.client.0.smithi172.stderr:1 2019-12-21T15:59:41.799 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: ceph osd dump 2019-12-21T15:59:41.799 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: grep 'osd.1 up' 2019-12-21T15:59:42.039 INFO:tasks.workunit.client.0.smithi172.stderr:osd.1 up in weight 1 up_from 32 up_thru 32 down_at 30 last_clean_interval [10,29) [v2:127.0.0.1:6810/131096,v1:127.0.0.1:6811/131096] [v2:127.0.0.1:6812/131096,v1:127.0.0.1:6813/131096] exists,up 9bb8026b-4cfb-486b-aa07-b00acd8d49cd 2019-12-21T15:59:42.039 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=0 2019-12-21T15:59:42.040 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: break 2019-12-21T15:59:42.040 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: return 0 2019-12-21T15:59:42.040 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1239: objectstore_tool: wait_for_clean 2019-12-21T15:59:42.040 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1548: wait_for_clean: local cmd= 2019-12-21T15:59:42.040 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1549: wait_for_clean: local num_active_clean=-1 2019-12-21T15:59:42.040 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1550: wait_for_clean: local cur_active_clean 2019-12-21T15:59:42.041 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1551: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2019-12-21T15:59:42.041 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1551: wait_for_clean: get_timeout_delays 90 .1 2019-12-21T15:59:42.041 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1494: get_timeout_delays: shopt -q -o xtrace 2019-12-21T15:59:42.041 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1494: get_timeout_delays: echo true 2019-12-21T15:59:42.041 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1494: get_timeout_delays: local trace=true 2019-12-21T15:59:42.041 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1495: get_timeout_delays: true 2019-12-21T15:59:42.042 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1495: get_timeout_delays: shopt -u -o xtrace 2019-12-21T15:59:42.131 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1551: wait_for_clean: local -a delays 2019-12-21T15:59:42.131 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1552: wait_for_clean: local -i loop=0 2019-12-21T15:59:42.131 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1554: wait_for_clean: flush_pg_stats 2019-12-21T15:59:42.131 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1977: flush_pg_stats: local timeout=300 2019-12-21T15:59:42.132 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1979: flush_pg_stats: ceph osd ls 2019-12-21T15:59:42.357 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1979: flush_pg_stats: ids='0 2019-12-21T15:59:42.358 INFO:tasks.workunit.client.0.smithi172.stderr:1 2019-12-21T15:59:42.358 INFO:tasks.workunit.client.0.smithi172.stderr:2' 2019-12-21T15:59:42.358 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1980: flush_pg_stats: seqs= 2019-12-21T15:59:42.358 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: flush_pg_stats: for osd in $ids 2019-12-21T15:59:42.358 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2019-12-21T15:59:42.468 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: seq=120259084292 2019-12-21T15:59:42.468 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: flush_pg_stats: seqs=' 0-120259084292' 2019-12-21T15:59:42.469 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: flush_pg_stats: for osd in $ids 2019-12-21T15:59:42.469 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2019-12-21T15:59:42.579 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: seq=137438953474 2019-12-21T15:59:42.579 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: flush_pg_stats: seqs=' 0-120259084292 1-137438953474' 2019-12-21T15:59:42.580 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: flush_pg_stats: for osd in $ids 2019-12-21T15:59:42.580 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2019-12-21T15:59:42.691 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: seq=64424509451 2019-12-21T15:59:42.691 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: flush_pg_stats: seqs=' 0-120259084292 1-137438953474 2-64424509451' 2019-12-21T15:59:42.691 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: flush_pg_stats: for s in $seqs 2019-12-21T15:59:42.692 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: echo 0-120259084292 2019-12-21T15:59:42.692 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: cut -d - -f 1 2019-12-21T15:59:42.693 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: osd=0 2019-12-21T15:59:42.693 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: echo 0-120259084292 2019-12-21T15:59:42.693 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: cut -d - -f 2 2019-12-21T15:59:42.694 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: seq=120259084292 2019-12-21T15:59:42.694 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1989: flush_pg_stats: echo 'waiting osd.0 seq 120259084292' 2019-12-21T15:59:42.694 INFO:tasks.workunit.client.0.smithi172.stderr:waiting osd.0 seq 120259084292 2019-12-21T15:59:42.694 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 0 2019-12-21T15:59:42.921 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 120259084291 -lt 120259084292 2019-12-21T15:59:42.921 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1991: flush_pg_stats: sleep 1 2019-12-21T15:59:43.922 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1992: flush_pg_stats: '[' 300 -eq 0 ']' 2019-12-21T15:59:43.923 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 0 2019-12-21T15:59:44.159 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 120259084291 -lt 120259084292 2019-12-21T15:59:44.160 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1991: flush_pg_stats: sleep 1 2019-12-21T15:59:45.160 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1992: flush_pg_stats: '[' 299 -eq 0 ']' 2019-12-21T15:59:45.161 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 0 2019-12-21T15:59:45.399 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 120259084293 -lt 120259084292 2019-12-21T15:59:45.399 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: flush_pg_stats: for s in $seqs 2019-12-21T15:59:45.399 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: echo 1-137438953474 2019-12-21T15:59:45.399 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: cut -d - -f 1 2019-12-21T15:59:45.400 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: osd=1 2019-12-21T15:59:45.400 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: echo 1-137438953474 2019-12-21T15:59:45.401 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: cut -d - -f 2 2019-12-21T15:59:45.401 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: seq=137438953474 2019-12-21T15:59:45.402 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1989: flush_pg_stats: echo 'waiting osd.1 seq 137438953474' 2019-12-21T15:59:45.402 INFO:tasks.workunit.client.0.smithi172.stderr:waiting osd.1 seq 137438953474 2019-12-21T15:59:45.402 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 1 2019-12-21T15:59:45.630 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 137438953474 -lt 137438953474 2019-12-21T15:59:45.630 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: flush_pg_stats: for s in $seqs 2019-12-21T15:59:45.630 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: echo 2-64424509451 2019-12-21T15:59:45.631 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: cut -d - -f 1 2019-12-21T15:59:45.631 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: osd=2 2019-12-21T15:59:45.632 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: echo 2-64424509451 2019-12-21T15:59:45.632 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: cut -d - -f 2 2019-12-21T15:59:45.633 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: seq=64424509451 2019-12-21T15:59:45.633 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1989: flush_pg_stats: echo 'waiting osd.2 seq 64424509451' 2019-12-21T15:59:45.633 INFO:tasks.workunit.client.0.smithi172.stderr:waiting osd.2 seq 64424509451 2019-12-21T15:59:45.633 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 2 2019-12-21T15:59:45.859 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 64424509451 -lt 64424509451 2019-12-21T15:59:45.859 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1555: wait_for_clean: get_num_pgs 2019-12-21T15:59:45.859 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: ceph --format json status 2019-12-21T15:59:45.860 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: jq .pgmap.num_pgs 2019-12-21T15:59:46.129 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1555: wait_for_clean: test 1 == 0 2019-12-21T15:59:46.129 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1559: wait_for_clean: true 2019-12-21T15:59:46.129 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1563: wait_for_clean: get_num_active_clean 2019-12-21T15:59:46.130 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: get_num_active_clean: local expression 2019-12-21T15:59:46.130 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2019-12-21T15:59:46.130 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("stale") | not)' 2019-12-21T15:59:46.130 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: ceph --format json pg dump pgs 2019-12-21T15:59:46.130 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2019-12-21T15:59:46.352 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1563: wait_for_clean: cur_active_clean=0 2019-12-21T15:59:46.353 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1564: wait_for_clean: get_num_pgs 2019-12-21T15:59:46.353 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: ceph --format json status 2019-12-21T15:59:46.353 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: jq .pgmap.num_pgs 2019-12-21T15:59:46.625 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1564: wait_for_clean: test 0 = 1 2019-12-21T15:59:46.625 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1565: wait_for_clean: test 0 '!=' -1 2019-12-21T15:59:46.625 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1566: wait_for_clean: loop=0 2019-12-21T15:59:46.625 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1567: wait_for_clean: num_active_clean=0 2019-12-21T15:59:46.625 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1575: wait_for_clean: eval 2019-12-21T15:59:46.625 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1576: wait_for_clean: sleep 0.1 2019-12-21T15:59:46.726 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1577: wait_for_clean: loop+=1 2019-12-21T15:59:46.726 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1559: wait_for_clean: true 2019-12-21T15:59:46.726 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1563: wait_for_clean: get_num_active_clean 2019-12-21T15:59:46.726 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: get_num_active_clean: local expression 2019-12-21T15:59:46.726 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2019-12-21T15:59:46.727 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("stale") | not)' 2019-12-21T15:59:46.727 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: ceph --format json pg dump pgs 2019-12-21T15:59:46.727 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2019-12-21T15:59:46.947 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1563: wait_for_clean: cur_active_clean=0 2019-12-21T15:59:46.947 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1564: wait_for_clean: get_num_pgs 2019-12-21T15:59:46.948 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: ceph --format json status 2019-12-21T15:59:46.948 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: jq .pgmap.num_pgs 2019-12-21T15:59:47.215 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1564: wait_for_clean: test 0 = 1 2019-12-21T15:59:47.215 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1565: wait_for_clean: test 0 '!=' 0 2019-12-21T15:59:47.215 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1568: wait_for_clean: get_is_making_recovery_progress 2019-12-21T15:59:47.215 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: get_is_making_recovery_progress: local recovery_progress 2019-12-21T15:59:47.215 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2019-12-21T15:59:47.216 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2019-12-21T15:59:47.216 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2019-12-21T15:59:47.216 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: ceph --format json status 2019-12-21T15:59:47.216 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2019-12-21T15:59:47.484 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: local progress=null 2019-12-21T15:59:47.484 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: test null '!=' null 2019-12-21T15:59:47.484 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1570: wait_for_clean: (( 1 >= 13 )) 2019-12-21T15:59:47.484 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1575: wait_for_clean: eval 2019-12-21T15:59:47.485 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1576: wait_for_clean: sleep 0.2 2019-12-21T15:59:47.685 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1577: wait_for_clean: loop+=1 2019-12-21T15:59:47.685 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1559: wait_for_clean: true 2019-12-21T15:59:47.686 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1563: wait_for_clean: get_num_active_clean 2019-12-21T15:59:47.686 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: get_num_active_clean: local expression 2019-12-21T15:59:47.686 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2019-12-21T15:59:47.686 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("stale") | not)' 2019-12-21T15:59:47.686 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: ceph --format json pg dump pgs 2019-12-21T15:59:47.686 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2019-12-21T15:59:47.906 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1563: wait_for_clean: cur_active_clean=0 2019-12-21T15:59:47.907 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1564: wait_for_clean: get_num_pgs 2019-12-21T15:59:47.907 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: ceph --format json status 2019-12-21T15:59:47.907 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: jq .pgmap.num_pgs 2019-12-21T15:59:48.175 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1564: wait_for_clean: test 0 = 1 2019-12-21T15:59:48.176 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1565: wait_for_clean: test 0 '!=' 0 2019-12-21T15:59:48.176 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1568: wait_for_clean: get_is_making_recovery_progress 2019-12-21T15:59:48.176 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: get_is_making_recovery_progress: local recovery_progress 2019-12-21T15:59:48.176 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2019-12-21T15:59:48.176 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2019-12-21T15:59:48.177 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2019-12-21T15:59:48.177 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: ceph --format json status 2019-12-21T15:59:48.177 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2019-12-21T15:59:48.448 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: local progress=null 2019-12-21T15:59:48.448 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: test null '!=' null 2019-12-21T15:59:48.448 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1570: wait_for_clean: (( 2 >= 13 )) 2019-12-21T15:59:48.448 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1575: wait_for_clean: eval 2019-12-21T15:59:48.448 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1576: wait_for_clean: sleep 0.4 2019-12-21T15:59:48.849 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1577: wait_for_clean: loop+=1 2019-12-21T15:59:48.849 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1559: wait_for_clean: true 2019-12-21T15:59:48.850 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1563: wait_for_clean: get_num_active_clean 2019-12-21T15:59:48.850 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: get_num_active_clean: local expression 2019-12-21T15:59:48.850 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2019-12-21T15:59:48.850 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("stale") | not)' 2019-12-21T15:59:48.850 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: ceph --format json pg dump pgs 2019-12-21T15:59:48.850 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2019-12-21T15:59:49.074 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1563: wait_for_clean: cur_active_clean=1 2019-12-21T15:59:49.075 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1564: wait_for_clean: get_num_pgs 2019-12-21T15:59:49.075 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: ceph --format json status 2019-12-21T15:59:49.075 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: jq .pgmap.num_pgs 2019-12-21T15:59:49.342 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1564: wait_for_clean: test 1 = 1 2019-12-21T15:59:49.342 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1564: wait_for_clean: break 2019-12-21T15:59:49.343 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: return 0 2019-12-21T15:59:49.343 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:470: TEST_auto_repair_bluestore_failed: get_pg testpool obj1 2019-12-21T15:59:49.343 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1029: get_pg: local poolname=testpool 2019-12-21T15:59:49.343 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1030: get_pg: local objectname=obj1 2019-12-21T15:59:49.343 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1032: get_pg: ceph --format json osd map testpool obj1 2019-12-21T15:59:49.343 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1032: get_pg: jq -r .pgid 2019-12-21T15:59:49.574 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:470: TEST_auto_repair_bluestore_failed: local pgid=1.0 2019-12-21T15:59:49.574 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:471: TEST_auto_repair_bluestore_failed: get_primary testpool obj1 2019-12-21T15:59:49.574 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_primary: local poolname=testpool 2019-12-21T15:59:49.574 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1133: get_primary: local objectname=obj1 2019-12-21T15:59:49.574 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1135: get_primary: ceph --format json osd map testpool obj1 2019-12-21T15:59:49.574 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1136: get_primary: jq .acting_primary 2019-12-21T15:59:49.805 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:471: TEST_auto_repair_bluestore_failed: local primary=1 2019-12-21T15:59:49.806 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:472: TEST_auto_repair_bluestore_failed: get_last_scrub_stamp 1.0 2019-12-21T15:59:49.806 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.0 2019-12-21T15:59:49.806 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:59:49.806 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:59:49.806 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2019-12-21T15:59:50.038 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:472: TEST_auto_repair_bluestore_failed: local last_scrub_stamp=2019-12-21T15:59:17.225700+0000 2019-12-21T15:59:50.038 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:473: TEST_auto_repair_bluestore_failed: ceph tell 1.0 deep_scrub 2019-12-21T15:59:50.139 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:59:50.139 INFO:tasks.workunit.client.0.smithi172.stdout: "deep": true, 2019-12-21T15:59:50.139 INFO:tasks.workunit.client.0.smithi172.stdout: "stamp": "2019-12-14T15:58:10.138191+0000" 2019-12-21T15:59:50.139 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:59:50.147 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:474: TEST_auto_repair_bluestore_failed: ceph tell 1.0 scrub 2019-12-21T15:59:50.253 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T15:59:50.253 INFO:tasks.workunit.client.0.smithi172.stdout: "deep": false, 2019-12-21T15:59:50.253 INFO:tasks.workunit.client.0.smithi172.stdout: "stamp": "2019-12-14T15:58:10.252239+0000" 2019-12-21T15:59:50.254 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T15:59:50.262 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:477: TEST_auto_repair_bluestore_failed: wait_for_scrub 1.0 2019-12-21T15:59:17.225700+0000 2019-12-21T15:59:50.262 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1789: wait_for_scrub: local pgid=1.0 2019-12-21T15:59:50.262 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1790: wait_for_scrub: local last_scrub=2019-12-21T15:59:17.225700+0000 2019-12-21T15:59:50.263 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1791: wait_for_scrub: local sname=last_scrub_stamp 2019-12-21T15:59:50.263 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i=0 )) 2019-12-21T15:59:50.263 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:59:50.263 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2019-12-21T15:59:50.263 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.0 2019-12-21T15:59:50.263 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:59:50.263 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:59:50.263 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2019-12-21T15:59:50.486 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:59:17.225700+0000 '>' 2019-12-21T15:59:17.225700+0000 2019-12-21T15:59:50.487 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:59:51.487 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:59:51.488 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:59:51.489 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2019-12-21T15:59:51.489 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.0 2019-12-21T15:59:51.489 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:59:51.489 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:59:51.489 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2019-12-21T15:59:51.721 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:59:17.225700+0000 '>' 2019-12-21T15:59:17.225700+0000 2019-12-21T15:59:51.721 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:59:52.722 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:59:52.722 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:59:52.723 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2019-12-21T15:59:52.723 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.0 2019-12-21T15:59:52.723 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:59:52.724 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:59:52.724 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2019-12-21T15:59:52.954 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-14T15:58:10.252239+0000 '>' 2019-12-21T15:59:17.225700+0000 2019-12-21T15:59:52.954 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:59:53.955 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:59:53.955 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:59:53.956 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2019-12-21T15:59:53.956 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.0 2019-12-21T15:59:53.956 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:59:53.956 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:59:53.956 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2019-12-21T15:59:54.189 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-14T15:58:10.252239+0000 '>' 2019-12-21T15:59:17.225700+0000 2019-12-21T15:59:54.189 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:59:55.190 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:59:55.190 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:59:55.191 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2019-12-21T15:59:55.191 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.0 2019-12-21T15:59:55.191 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:59:55.191 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:59:55.191 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2019-12-21T15:59:55.423 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-14T15:58:10.252239+0000 '>' 2019-12-21T15:59:17.225700+0000 2019-12-21T15:59:55.424 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:59:56.424 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:59:56.425 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:59:56.425 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2019-12-21T15:59:56.426 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.0 2019-12-21T15:59:56.426 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:59:56.426 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:59:56.426 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2019-12-21T15:59:56.682 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-14T15:58:10.252239+0000 '>' 2019-12-21T15:59:17.225700+0000 2019-12-21T15:59:56.683 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:59:57.684 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:59:57.684 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:59:57.684 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2019-12-21T15:59:57.685 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.0 2019-12-21T15:59:57.685 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:59:57.685 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:59:57.685 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2019-12-21T15:59:57.920 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-14T15:58:10.252239+0000 '>' 2019-12-21T15:59:17.225700+0000 2019-12-21T15:59:57.920 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T15:59:58.921 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T15:59:58.921 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T15:59:58.922 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2019-12-21T15:59:58.922 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.0 2019-12-21T15:59:58.922 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T15:59:58.922 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T15:59:58.922 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2019-12-21T15:59:59.156 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-14T15:58:10.252239+0000 '>' 2019-12-21T15:59:17.225700+0000 2019-12-21T15:59:59.157 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T16:00:00.155 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T16:00:00.155 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T16:00:00.156 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2019-12-21T16:00:00.156 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.0 2019-12-21T16:00:00.157 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T16:00:00.157 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T16:00:00.157 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2019-12-21T16:00:00.387 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-14T15:58:10.252239+0000 '>' 2019-12-21T15:59:17.225700+0000 2019-12-21T16:00:00.388 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T16:00:01.389 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T16:00:01.389 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T16:00:01.390 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2019-12-21T16:00:01.390 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.0 2019-12-21T16:00:01.390 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T16:00:01.390 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T16:00:01.390 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2019-12-21T16:00:01.624 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-14T15:58:10.252239+0000 '>' 2019-12-21T15:59:17.225700+0000 2019-12-21T16:00:01.624 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T16:00:02.624 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T16:00:02.625 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T16:00:02.625 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2019-12-21T16:00:02.626 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.0 2019-12-21T16:00:02.626 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T16:00:02.626 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T16:00:02.626 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2019-12-21T16:00:02.854 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:59:58.064980+0000 '>' 2019-12-21T15:59:17.225700+0000 2019-12-21T16:00:02.854 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1795: wait_for_scrub: return 0 2019-12-21T16:00:02.854 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:478: TEST_auto_repair_bluestore_failed: wait_for_clean 2019-12-21T16:00:02.854 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1548: wait_for_clean: local cmd= 2019-12-21T16:00:02.855 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1549: wait_for_clean: local num_active_clean=-1 2019-12-21T16:00:02.855 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1550: wait_for_clean: local cur_active_clean 2019-12-21T16:00:02.855 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1551: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2019-12-21T16:00:02.855 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1551: wait_for_clean: get_timeout_delays 90 .1 2019-12-21T16:00:02.855 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1494: get_timeout_delays: shopt -q -o xtrace 2019-12-21T16:00:02.855 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1494: get_timeout_delays: echo true 2019-12-21T16:00:02.856 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1494: get_timeout_delays: local trace=true 2019-12-21T16:00:02.856 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1495: get_timeout_delays: true 2019-12-21T16:00:02.856 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1495: get_timeout_delays: shopt -u -o xtrace 2019-12-21T16:00:02.946 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1551: wait_for_clean: local -a delays 2019-12-21T16:00:02.947 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1552: wait_for_clean: local -i loop=0 2019-12-21T16:00:02.947 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1554: wait_for_clean: flush_pg_stats 2019-12-21T16:00:02.947 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1977: flush_pg_stats: local timeout=300 2019-12-21T16:00:02.947 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1979: flush_pg_stats: ceph osd ls 2019-12-21T16:00:03.172 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1979: flush_pg_stats: ids='0 2019-12-21T16:00:03.172 INFO:tasks.workunit.client.0.smithi172.stderr:1 2019-12-21T16:00:03.172 INFO:tasks.workunit.client.0.smithi172.stderr:2' 2019-12-21T16:00:03.172 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1980: flush_pg_stats: seqs= 2019-12-21T16:00:03.173 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: flush_pg_stats: for osd in $ids 2019-12-21T16:00:03.173 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2019-12-21T16:00:03.282 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: seq=120259084297 2019-12-21T16:00:03.283 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: flush_pg_stats: seqs=' 0-120259084297' 2019-12-21T16:00:03.283 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: flush_pg_stats: for osd in $ids 2019-12-21T16:00:03.283 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2019-12-21T16:00:03.393 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: seq=137438953479 2019-12-21T16:00:03.394 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: flush_pg_stats: seqs=' 0-120259084297 1-137438953479' 2019-12-21T16:00:03.394 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: flush_pg_stats: for osd in $ids 2019-12-21T16:00:03.394 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2019-12-21T16:00:03.504 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: seq=64424509456 2019-12-21T16:00:03.504 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: flush_pg_stats: seqs=' 0-120259084297 1-137438953479 2-64424509456' 2019-12-21T16:00:03.504 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: flush_pg_stats: for s in $seqs 2019-12-21T16:00:03.505 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: echo 0-120259084297 2019-12-21T16:00:03.505 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: cut -d - -f 1 2019-12-21T16:00:03.505 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: osd=0 2019-12-21T16:00:03.506 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: echo 0-120259084297 2019-12-21T16:00:03.506 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: cut -d - -f 2 2019-12-21T16:00:03.507 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: seq=120259084297 2019-12-21T16:00:03.507 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1989: flush_pg_stats: echo 'waiting osd.0 seq 120259084297' 2019-12-21T16:00:03.507 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 0 2019-12-21T16:00:03.507 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.0 seq 120259084297 2019-12-21T16:00:03.737 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 120259084296 -lt 120259084297 2019-12-21T16:00:03.737 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1991: flush_pg_stats: sleep 1 2019-12-21T16:00:04.735 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1992: flush_pg_stats: '[' 300 -eq 0 ']' 2019-12-21T16:00:04.736 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 0 2019-12-21T16:00:04.972 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 120259084298 -lt 120259084297 2019-12-21T16:00:04.972 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: flush_pg_stats: for s in $seqs 2019-12-21T16:00:04.973 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: echo 1-137438953479 2019-12-21T16:00:04.973 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: cut -d - -f 1 2019-12-21T16:00:04.973 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: osd=1 2019-12-21T16:00:04.974 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: echo 1-137438953479 2019-12-21T16:00:04.974 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: cut -d - -f 2 2019-12-21T16:00:04.975 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: seq=137438953479 2019-12-21T16:00:04.975 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1989: flush_pg_stats: echo 'waiting osd.1 seq 137438953479' 2019-12-21T16:00:04.976 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 1 2019-12-21T16:00:04.976 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.1 seq 137438953479 2019-12-21T16:00:05.202 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 137438953479 -lt 137438953479 2019-12-21T16:00:05.202 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: flush_pg_stats: for s in $seqs 2019-12-21T16:00:05.203 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: echo 2-64424509456 2019-12-21T16:00:05.203 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: cut -d - -f 1 2019-12-21T16:00:05.204 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: osd=2 2019-12-21T16:00:05.204 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: echo 2-64424509456 2019-12-21T16:00:05.204 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: cut -d - -f 2 2019-12-21T16:00:05.205 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: seq=64424509456 2019-12-21T16:00:05.205 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1989: flush_pg_stats: echo 'waiting osd.2 seq 64424509456' 2019-12-21T16:00:05.205 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 2 2019-12-21T16:00:05.205 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.2 seq 64424509456 2019-12-21T16:00:05.432 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 64424509456 -lt 64424509456 2019-12-21T16:00:05.433 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1555: wait_for_clean: get_num_pgs 2019-12-21T16:00:05.433 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: ceph --format json status 2019-12-21T16:00:05.434 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: jq .pgmap.num_pgs 2019-12-21T16:00:05.702 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1555: wait_for_clean: test 1 == 0 2019-12-21T16:00:05.702 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1559: wait_for_clean: true 2019-12-21T16:00:05.702 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1563: wait_for_clean: get_num_active_clean 2019-12-21T16:00:05.703 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: get_num_active_clean: local expression 2019-12-21T16:00:05.703 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2019-12-21T16:00:05.703 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("stale") | not)' 2019-12-21T16:00:05.703 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: ceph --format json pg dump pgs 2019-12-21T16:00:05.703 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2019-12-21T16:00:05.925 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1563: wait_for_clean: cur_active_clean=1 2019-12-21T16:00:05.925 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1564: wait_for_clean: get_num_pgs 2019-12-21T16:00:05.925 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: ceph --format json status 2019-12-21T16:00:05.925 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: jq .pgmap.num_pgs 2019-12-21T16:00:06.193 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1564: wait_for_clean: test 1 = 1 2019-12-21T16:00:06.193 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1564: wait_for_clean: break 2019-12-21T16:00:06.194 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: return 0 2019-12-21T16:00:06.194 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:479: TEST_auto_repair_bluestore_failed: flush_pg_stats 2019-12-21T16:00:06.194 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1977: flush_pg_stats: local timeout=300 2019-12-21T16:00:06.194 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1979: flush_pg_stats: ceph osd ls 2019-12-21T16:00:06.420 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1979: flush_pg_stats: ids='0 2019-12-21T16:00:06.420 INFO:tasks.workunit.client.0.smithi172.stderr:1 2019-12-21T16:00:06.421 INFO:tasks.workunit.client.0.smithi172.stderr:2' 2019-12-21T16:00:06.421 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1980: flush_pg_stats: seqs= 2019-12-21T16:00:06.421 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: flush_pg_stats: for osd in $ids 2019-12-21T16:00:06.421 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2019-12-21T16:00:06.531 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: seq=120259084299 2019-12-21T16:00:06.531 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: flush_pg_stats: seqs=' 0-120259084299' 2019-12-21T16:00:06.531 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: flush_pg_stats: for osd in $ids 2019-12-21T16:00:06.532 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2019-12-21T16:00:06.646 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: seq=137438953480 2019-12-21T16:00:06.646 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: flush_pg_stats: seqs=' 0-120259084299 1-137438953480' 2019-12-21T16:00:06.646 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: flush_pg_stats: for osd in $ids 2019-12-21T16:00:06.647 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2019-12-21T16:00:06.758 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: seq=64424509458 2019-12-21T16:00:06.759 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: flush_pg_stats: seqs=' 0-120259084299 1-137438953480 2-64424509458' 2019-12-21T16:00:06.759 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: flush_pg_stats: for s in $seqs 2019-12-21T16:00:06.759 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: echo 0-120259084299 2019-12-21T16:00:06.759 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: cut -d - -f 1 2019-12-21T16:00:06.760 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: osd=0 2019-12-21T16:00:06.760 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: echo 0-120259084299 2019-12-21T16:00:06.760 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: cut -d - -f 2 2019-12-21T16:00:06.761 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: seq=120259084299 2019-12-21T16:00:06.761 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1989: flush_pg_stats: echo 'waiting osd.0 seq 120259084299' 2019-12-21T16:00:06.762 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 0 2019-12-21T16:00:06.762 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.0 seq 120259084299 2019-12-21T16:00:06.985 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 120259084298 -lt 120259084299 2019-12-21T16:00:06.985 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1991: flush_pg_stats: sleep 1 2019-12-21T16:00:07.986 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1992: flush_pg_stats: '[' 300 -eq 0 ']' 2019-12-21T16:00:07.987 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 0 2019-12-21T16:00:08.214 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 120259084298 -lt 120259084299 2019-12-21T16:00:08.215 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1991: flush_pg_stats: sleep 1 2019-12-21T16:00:09.216 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1992: flush_pg_stats: '[' 299 -eq 0 ']' 2019-12-21T16:00:09.216 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 0 2019-12-21T16:00:09.451 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 120259084299 -lt 120259084299 2019-12-21T16:00:09.451 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: flush_pg_stats: for s in $seqs 2019-12-21T16:00:09.451 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: echo 1-137438953480 2019-12-21T16:00:09.452 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: cut -d - -f 1 2019-12-21T16:00:09.452 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: osd=1 2019-12-21T16:00:09.452 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: echo 1-137438953480 2019-12-21T16:00:09.453 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: cut -d - -f 2 2019-12-21T16:00:09.453 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: seq=137438953480 2019-12-21T16:00:09.454 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1989: flush_pg_stats: echo 'waiting osd.1 seq 137438953480' 2019-12-21T16:00:09.454 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 1 2019-12-21T16:00:09.454 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.1 seq 137438953480 2019-12-21T16:00:09.681 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 137438953481 -lt 137438953480 2019-12-21T16:00:09.682 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: flush_pg_stats: for s in $seqs 2019-12-21T16:00:09.682 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: echo 2-64424509458 2019-12-21T16:00:09.682 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: cut -d - -f 1 2019-12-21T16:00:09.683 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: osd=2 2019-12-21T16:00:09.683 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: echo 2-64424509458 2019-12-21T16:00:09.684 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: cut -d - -f 2 2019-12-21T16:00:09.684 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: seq=64424509458 2019-12-21T16:00:09.684 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1989: flush_pg_stats: echo 'waiting osd.2 seq 64424509458' 2019-12-21T16:00:09.685 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 2 2019-12-21T16:00:09.685 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.2 seq 64424509458 2019-12-21T16:00:09.911 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 64424509458 -lt 64424509458 2019-12-21T16:00:09.911 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:480: TEST_auto_repair_bluestore_failed: grep scrub_finish td/osd-scrub-repair/osd.1.log 2019-12-21T16:00:09.912 INFO:tasks.workunit.client.0.smithi172.stdout:2019-12-21T15:59:57.837+0000 7fe9ef9bb700 20 osd.1 pg_epoch: 33 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=32/33 n=10 ec=17/17 lis/c=32/32 les/c/f=33/33/0 sis=32) [1,0] r=0 lpr=32 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+repair AUTO_REPAIR] scrub_finish 2019-12-21T16:00:09.913 INFO:tasks.workunit.client.0.smithi172.stdout:2019-12-21T15:59:57.837+0000 7fe9ef9bb700 20 osd.1 pg_epoch: 33 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=32/33 n=10 ec=17/17 lis/c=32/32 les/c/f=33/33/0 sis=32) [1,0] r=0 lpr=32 crt=21'10 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair AUTO_REPAIR mbc={255={(1+0)=1}}] scrub_finish Set scrub_after_recovery 2019-12-21T16:00:09.913 INFO:tasks.workunit.client.0.smithi172.stdout:2019-12-21T15:59:58.063+0000 7fe9ef9bb700 20 osd.1 pg_epoch: 33 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=32/33 n=10 ec=17/17 lis/c=32/32 les/c/f=33/33/0 sis=32) [1,0] r=0 lpr=32 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent CHECK_REPAIR] scrub_finish 2019-12-21T16:00:09.913 INFO:tasks.workunit.client.0.smithi172.stdout:2019-12-21T15:59:58.063+0000 7fe9ef9bb700 10 osd.1 pg_epoch: 33 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=32/33 n=10 ec=17/17 lis/c=32/32 les/c/f=33/33/0 sis=32) [1,0] r=0 lpr=32 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent+failed_repair] scrub_finish 3 error(s) still present after re-scrub 2019-12-21T16:00:09.913 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:481: TEST_auto_repair_bluestore_failed: grep -q 'scrub_finish.*still present after re-scrub' td/osd-scrub-repair/osd.1.log 2019-12-21T16:00:09.914 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:482: TEST_auto_repair_bluestore_failed: ceph pg dump pgs 2019-12-21T16:00:10.126 INFO:tasks.workunit.client.0.smithi172.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2019-12-21T16:00:10.126 INFO:tasks.workunit.client.0.smithi172.stdout:1.0 10 0 0 0 0 63 0 0 10 10 active+clean+inconsistent+failed_repair 2019-12-21T15:59:58.065017+0000 21'10 33:88 [1,0] 1 [1,0] 1 21'10 2019-12-21T15:59:58.064980+0000 21'10 2019-12-21T15:59:58.064980+0000 0 2019-12-21T16:00:10.126 INFO:tasks.workunit.client.0.smithi172.stdout: 2019-12-21T16:00:10.126 INFO:tasks.workunit.client.0.smithi172.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilisation. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2019-12-21T16:00:10.128 INFO:tasks.workunit.client.0.smithi172.stderr:dumped pgs 2019-12-21T16:00:10.135 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:483: TEST_auto_repair_bluestore_failed: ceph pg dump pgs 2019-12-21T16:00:10.136 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:483: TEST_auto_repair_bluestore_failed: pgid 2019-12-21T16:00:10.136 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh: line 483: pgid: command not found 2019-12-21T16:00:10.136 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:483: TEST_auto_repair_bluestore_failed: grep -q '^.*+failed_repair' 2019-12-21T16:00:10.347 INFO:tasks.workunit.client.0.smithi172.stderr:dumped pgs 2019-12-21T16:00:10.356 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:487: TEST_auto_repair_bluestore_failed: get_not_primary testpool obj1 2019-12-21T16:00:10.356 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1165: get_not_primary: local poolname=testpool 2019-12-21T16:00:10.356 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1166: get_not_primary: local objectname=obj1 2019-12-21T16:00:10.356 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: get_not_primary: get_primary testpool obj1 2019-12-21T16:00:10.356 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_primary: local poolname=testpool 2019-12-21T16:00:10.357 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1133: get_primary: local objectname=obj1 2019-12-21T16:00:10.357 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1135: get_primary: ceph --format json osd map testpool obj1 2019-12-21T16:00:10.357 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1136: get_primary: jq .acting_primary 2019-12-21T16:00:10.587 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: get_not_primary: local primary=1 2019-12-21T16:00:10.587 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1169: get_not_primary: ceph --format json osd map testpool obj1 2019-12-21T16:00:10.588 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1170: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2019-12-21T16:00:10.819 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:487: TEST_auto_repair_bluestore_failed: objectstore_tool td/osd-scrub-repair 0 obj1 list-attrs 2019-12-21T16:00:10.819 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: objectstore_tool: local dir=td/osd-scrub-repair 2019-12-21T16:00:10.819 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: objectstore_tool: shift 2019-12-21T16:00:10.819 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1235: objectstore_tool: local id=0 2019-12-21T16:00:10.819 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1236: objectstore_tool: shift 2019-12-21T16:00:10.819 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1238: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 obj1 list-attrs 2019-12-21T16:00:10.820 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1205: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2019-12-21T16:00:10.820 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nowait: shift 2019-12-21T16:00:10.820 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1207: _objectstore_tool_nowait: local id=0 2019-12-21T16:00:10.820 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1208: _objectstore_tool_nowait: shift 2019-12-21T16:00:10.820 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1210: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2019-12-21T16:00:10.820 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2019-12-21T16:00:10.821 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2019-12-21T16:00:10.821 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2019-12-21T16:00:10.821 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2019-12-21T16:00:10.821 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2019-12-21T16:00:10.923 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2019-12-21T16:00:10.924 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1212: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 obj1 list-attrs 2019-12-21T16:00:10.924 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1193: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2019-12-21T16:00:10.924 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1194: _objectstore_tool_nodown: shift 2019-12-21T16:00:10.924 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1195: _objectstore_tool_nodown: local id=0 2019-12-21T16:00:10.924 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: _objectstore_tool_nodown: shift 2019-12-21T16:00:10.924 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2019-12-21T16:00:10.924 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj1 list-attrs 2019-12-21T16:00:11.489 INFO:tasks.workunit.client.0.smithi172.stdout:_ 2019-12-21T16:00:11.489 INFO:tasks.workunit.client.0.smithi172.stdout:snapset 2019-12-21T16:00:12.008 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1213: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2019-12-21T16:00:12.008 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:842: activate_osd: local dir=td/osd-scrub-repair 2019-12-21T16:00:12.008 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: shift 2019-12-21T16:00:12.009 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: local id=0 2019-12-21T16:00:12.009 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: shift 2019-12-21T16:00:12.009 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local osd_data=td/osd-scrub-repair/0 2019-12-21T16:00:12.009 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local 'ceph_args=--fsid=973e3fc1-e63e-4bde-9db6-84c68fd5e93e --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2019-12-21T16:00:12.009 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2019-12-21T16:00:12.009 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-journal-size=100' 2019-12-21T16:00:12.009 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2019-12-21T16:00:12.010 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2019-12-21T16:00:12.010 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2019-12-21T16:00:12.010 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --chdir=' 2019-12-21T16:00:12.010 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+= 2019-12-21T16:00:12.010 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2019-12-21T16:00:12.010 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: get_asok_path 2019-12-21T16:00:12.010 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2019-12-21T16:00:12.011 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2019-12-21T16:00:12.011 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2019-12-21T16:00:12.011 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T16:00:12.011 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.119575 2019-12-21T16:00:12.011 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.119575/$cluster-$name.asok' 2019-12-21T16:00:12.011 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.119575/$cluster-$name.asok' 2019-12-21T16:00:12.011 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --debug-osd=20' 2019-12-21T16:00:12.012 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2019-12-21T16:00:12.012 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2019-12-21T16:00:12.012 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2019-12-21T16:00:12.012 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2019-12-21T16:00:12.012 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2019-12-21T16:00:12.012 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' ' 2019-12-21T16:00:12.012 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2019-12-21T16:00:12.013 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: mkdir -p td/osd-scrub-repair/0 2019-12-21T16:00:12.013 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: echo start osd.0 2019-12-21T16:00:12.013 INFO:tasks.workunit.client.0.smithi172.stderr:start osd.0 2019-12-21T16:00:12.013 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph-osd -i 0 --fsid=973e3fc1-e63e-4bde-9db6-84c68fd5e93e --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.119575/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2019-12-21T16:00:12.013 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: cat td/osd-scrub-repair/0/whoami 2019-12-21T16:00:12.013 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: '[' 0 = 0 ']' 2019-12-21T16:00:12.014 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph osd dump --format=json 2019-12-21T16:00:12.014 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: jq '.flags_set[]' 2019-12-21T16:00:12.014 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: grep -q '"noup"' 2019-12-21T16:00:12.029 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:00:12.028+0000 7fb1bdf57ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T16:00:12.031 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:00:12.029+0000 7fb1bdf57ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T16:00:12.033 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:00:12.031+0000 7fb1bdf57ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T16:00:12.239 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: wait_for_osd up 0 2019-12-21T16:00:12.239 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:914: wait_for_osd: local state=up 2019-12-21T16:00:12.239 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local id=0 2019-12-21T16:00:12.239 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: status=1 2019-12-21T16:00:12.239 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i=0 )) 2019-12-21T16:00:12.239 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i < 300 )) 2019-12-21T16:00:12.240 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: echo 0 2019-12-21T16:00:12.240 INFO:tasks.workunit.client.0.smithi172.stderr:0 2019-12-21T16:00:12.240 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: ceph osd dump 2019-12-21T16:00:12.240 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: grep 'osd.0 up' 2019-12-21T16:00:12.482 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: sleep 1 2019-12-21T16:00:12.552 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:00:12.550+0000 7fb1bdf57ec0 -1 Falling back to public interface 2019-12-21T16:00:13.279 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:00:13.277+0000 7fb1bdf57ec0 -1 osd.0 33 log_to_monitors {default=true} 2019-12-21T16:00:13.483 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i++ )) 2019-12-21T16:00:13.484 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i < 300 )) 2019-12-21T16:00:13.484 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: echo 1 2019-12-21T16:00:13.484 INFO:tasks.workunit.client.0.smithi172.stderr:1 2019-12-21T16:00:13.484 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: grep 'osd.0 up' 2019-12-21T16:00:13.484 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: ceph osd dump 2019-12-21T16:00:13.712 INFO:tasks.workunit.client.0.smithi172.stderr:osd.0 up in weight 1 up_from 36 up_thru 30 down_at 34 last_clean_interval [28,33) [v2:127.0.0.1:6802/133202,v1:127.0.0.1:6803/133202] [v2:127.0.0.1:6804/133202,v1:127.0.0.1:6805/133202] exists,up 4236b440-d79c-4b13-9ee0-6323732a2f6e 2019-12-21T16:00:13.712 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=0 2019-12-21T16:00:13.712 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: break 2019-12-21T16:00:13.713 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: return 0 2019-12-21T16:00:13.713 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1239: objectstore_tool: wait_for_clean 2019-12-21T16:00:13.713 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1548: wait_for_clean: local cmd= 2019-12-21T16:00:13.713 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1549: wait_for_clean: local num_active_clean=-1 2019-12-21T16:00:13.713 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1550: wait_for_clean: local cur_active_clean 2019-12-21T16:00:13.713 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1551: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2019-12-21T16:00:13.714 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1551: wait_for_clean: get_timeout_delays 90 .1 2019-12-21T16:00:13.714 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1494: get_timeout_delays: shopt -q -o xtrace 2019-12-21T16:00:13.714 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1494: get_timeout_delays: echo true 2019-12-21T16:00:13.714 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1494: get_timeout_delays: local trace=true 2019-12-21T16:00:13.714 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1495: get_timeout_delays: true 2019-12-21T16:00:13.714 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1495: get_timeout_delays: shopt -u -o xtrace 2019-12-21T16:00:13.805 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1551: wait_for_clean: local -a delays 2019-12-21T16:00:13.805 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1552: wait_for_clean: local -i loop=0 2019-12-21T16:00:13.805 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1554: wait_for_clean: flush_pg_stats 2019-12-21T16:00:13.805 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1977: flush_pg_stats: local timeout=300 2019-12-21T16:00:13.805 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1979: flush_pg_stats: ceph osd ls 2019-12-21T16:00:14.037 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1979: flush_pg_stats: ids='0 2019-12-21T16:00:14.037 INFO:tasks.workunit.client.0.smithi172.stderr:1 2019-12-21T16:00:14.037 INFO:tasks.workunit.client.0.smithi172.stderr:2' 2019-12-21T16:00:14.038 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1980: flush_pg_stats: seqs= 2019-12-21T16:00:14.038 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: flush_pg_stats: for osd in $ids 2019-12-21T16:00:14.038 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2019-12-21T16:00:14.146 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: seq=154618822658 2019-12-21T16:00:14.146 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: flush_pg_stats: seqs=' 0-154618822658' 2019-12-21T16:00:14.146 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: flush_pg_stats: for osd in $ids 2019-12-21T16:00:14.147 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2019-12-21T16:00:14.258 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: seq=137438953483 2019-12-21T16:00:14.258 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: flush_pg_stats: seqs=' 0-154618822658 1-137438953483' 2019-12-21T16:00:14.258 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: flush_pg_stats: for osd in $ids 2019-12-21T16:00:14.258 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2019-12-21T16:00:14.368 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: seq=64424509460 2019-12-21T16:00:14.368 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: flush_pg_stats: seqs=' 0-154618822658 1-137438953483 2-64424509460' 2019-12-21T16:00:14.369 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: flush_pg_stats: for s in $seqs 2019-12-21T16:00:14.369 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: echo 0-154618822658 2019-12-21T16:00:14.369 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: cut -d - -f 1 2019-12-21T16:00:14.370 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: osd=0 2019-12-21T16:00:14.370 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: echo 0-154618822658 2019-12-21T16:00:14.370 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: cut -d - -f 2 2019-12-21T16:00:14.371 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: seq=154618822658 2019-12-21T16:00:14.371 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1989: flush_pg_stats: echo 'waiting osd.0 seq 154618822658' 2019-12-21T16:00:14.371 INFO:tasks.workunit.client.0.smithi172.stderr:waiting osd.0 seq 154618822658 2019-12-21T16:00:14.372 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 0 2019-12-21T16:00:14.598 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 154618822658 -lt 154618822658 2019-12-21T16:00:14.599 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: flush_pg_stats: for s in $seqs 2019-12-21T16:00:14.599 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: echo 1-137438953483 2019-12-21T16:00:14.599 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: cut -d - -f 1 2019-12-21T16:00:14.600 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: osd=1 2019-12-21T16:00:14.600 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: echo 1-137438953483 2019-12-21T16:00:14.601 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: cut -d - -f 2 2019-12-21T16:00:14.601 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: seq=137438953483 2019-12-21T16:00:14.602 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1989: flush_pg_stats: echo 'waiting osd.1 seq 137438953483' 2019-12-21T16:00:14.602 INFO:tasks.workunit.client.0.smithi172.stderr:waiting osd.1 seq 137438953483 2019-12-21T16:00:14.602 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 1 2019-12-21T16:00:14.847 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 137438953483 -lt 137438953483 2019-12-21T16:00:14.847 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: flush_pg_stats: for s in $seqs 2019-12-21T16:00:14.847 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: echo 2-64424509460 2019-12-21T16:00:14.848 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: cut -d - -f 1 2019-12-21T16:00:14.848 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: osd=2 2019-12-21T16:00:14.849 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: echo 2-64424509460 2019-12-21T16:00:14.849 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: cut -d - -f 2 2019-12-21T16:00:14.850 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: seq=64424509460 2019-12-21T16:00:14.850 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1989: flush_pg_stats: echo 'waiting osd.2 seq 64424509460' 2019-12-21T16:00:14.850 INFO:tasks.workunit.client.0.smithi172.stderr:waiting osd.2 seq 64424509460 2019-12-21T16:00:14.851 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 2 2019-12-21T16:00:15.079 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 64424509460 -lt 64424509460 2019-12-21T16:00:15.079 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1555: wait_for_clean: get_num_pgs 2019-12-21T16:00:15.080 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: ceph --format json status 2019-12-21T16:00:15.080 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: jq .pgmap.num_pgs 2019-12-21T16:00:15.351 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1555: wait_for_clean: test 1 == 0 2019-12-21T16:00:15.351 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1559: wait_for_clean: true 2019-12-21T16:00:15.351 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1563: wait_for_clean: get_num_active_clean 2019-12-21T16:00:15.352 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: get_num_active_clean: local expression 2019-12-21T16:00:15.352 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2019-12-21T16:00:15.352 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("stale") | not)' 2019-12-21T16:00:15.352 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: ceph --format json pg dump pgs 2019-12-21T16:00:15.352 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2019-12-21T16:00:15.573 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1563: wait_for_clean: cur_active_clean=1 2019-12-21T16:00:15.574 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1564: wait_for_clean: get_num_pgs 2019-12-21T16:00:15.574 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: ceph --format json status 2019-12-21T16:00:15.574 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: jq .pgmap.num_pgs 2019-12-21T16:00:15.842 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1564: wait_for_clean: test 1 = 1 2019-12-21T16:00:15.842 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1564: wait_for_clean: break 2019-12-21T16:00:15.843 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: return 0 2019-12-21T16:00:15.843 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:488: TEST_auto_repair_bluestore_failed: rados --pool testpool get obj1 td/osd-scrub-repair/COPY 2019-12-21T16:00:15.864 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:489: TEST_auto_repair_bluestore_failed: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2019-12-21T16:00:15.865 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:490: TEST_auto_repair_bluestore_failed: grep scrub_finish td/osd-scrub-repair/osd.1.log 2019-12-21T16:00:15.867 INFO:tasks.workunit.client.0.smithi172.stdout:2019-12-21T15:59:57.837+0000 7fe9ef9bb700 20 osd.1 pg_epoch: 33 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=32/33 n=10 ec=17/17 lis/c=32/32 les/c/f=33/33/0 sis=32) [1,0] r=0 lpr=32 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+repair AUTO_REPAIR] scrub_finish 2019-12-21T16:00:15.867 INFO:tasks.workunit.client.0.smithi172.stdout:2019-12-21T15:59:57.837+0000 7fe9ef9bb700 20 osd.1 pg_epoch: 33 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=32/33 n=10 ec=17/17 lis/c=32/32 les/c/f=33/33/0 sis=32) [1,0] r=0 lpr=32 crt=21'10 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair AUTO_REPAIR mbc={255={(1+0)=1}}] scrub_finish Set scrub_after_recovery 2019-12-21T16:00:15.867 INFO:tasks.workunit.client.0.smithi172.stdout:2019-12-21T15:59:58.063+0000 7fe9ef9bb700 20 osd.1 pg_epoch: 33 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=32/33 n=10 ec=17/17 lis/c=32/32 les/c/f=33/33/0 sis=32) [1,0] r=0 lpr=32 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent CHECK_REPAIR] scrub_finish 2019-12-21T16:00:15.867 INFO:tasks.workunit.client.0.smithi172.stdout:2019-12-21T15:59:58.063+0000 7fe9ef9bb700 10 osd.1 pg_epoch: 33 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=32/33 n=10 ec=17/17 lis/c=32/32 les/c/f=33/33/0 sis=32) [1,0] r=0 lpr=32 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent+failed_repair] scrub_finish 3 error(s) still present after re-scrub 2019-12-21T16:00:15.868 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:493: TEST_auto_repair_bluestore_failed: get_primary testpool SOMETHING 2019-12-21T16:00:15.868 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_primary: local poolname=testpool 2019-12-21T16:00:15.868 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1133: get_primary: local objectname=SOMETHING 2019-12-21T16:00:15.869 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1135: get_primary: ceph --format json osd map testpool SOMETHING 2019-12-21T16:00:15.869 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1136: get_primary: jq .acting_primary 2019-12-21T16:00:16.099 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:493: TEST_auto_repair_bluestore_failed: objectstore_tool td/osd-scrub-repair 1 obj2 remove 2019-12-21T16:00:16.099 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: objectstore_tool: local dir=td/osd-scrub-repair 2019-12-21T16:00:16.100 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: objectstore_tool: shift 2019-12-21T16:00:16.100 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1235: objectstore_tool: local id=1 2019-12-21T16:00:16.100 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1236: objectstore_tool: shift 2019-12-21T16:00:16.100 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1238: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 obj2 remove 2019-12-21T16:00:16.100 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1205: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2019-12-21T16:00:16.100 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nowait: shift 2019-12-21T16:00:16.100 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1207: _objectstore_tool_nowait: local id=1 2019-12-21T16:00:16.101 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1208: _objectstore_tool_nowait: shift 2019-12-21T16:00:16.101 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1210: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2019-12-21T16:00:16.101 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2019-12-21T16:00:16.101 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2019-12-21T16:00:16.101 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2019-12-21T16:00:16.101 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2019-12-21T16:00:16.101 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2019-12-21T16:00:16.205 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2019-12-21T16:00:16.205 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1212: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 obj2 remove 2019-12-21T16:00:16.206 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1193: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2019-12-21T16:00:16.206 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1194: _objectstore_tool_nodown: shift 2019-12-21T16:00:16.206 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1195: _objectstore_tool_nodown: local id=1 2019-12-21T16:00:16.206 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: _objectstore_tool_nodown: shift 2019-12-21T16:00:16.206 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2019-12-21T16:00:16.207 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj2 remove 2019-12-21T16:00:16.798 INFO:tasks.workunit.client.0.smithi172.stderr:Error getting attr on : 1.0_head,#1:104778fc:::obj2:head#, (61) No data available 2019-12-21T16:00:16.799 INFO:tasks.workunit.client.0.smithi172.stdout:remove #1:104778fc:::obj2:head# 2019-12-21T16:00:17.320 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1213: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2019-12-21T16:00:17.320 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:842: activate_osd: local dir=td/osd-scrub-repair 2019-12-21T16:00:17.320 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: shift 2019-12-21T16:00:17.320 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: local id=1 2019-12-21T16:00:17.320 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: shift 2019-12-21T16:00:17.320 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local osd_data=td/osd-scrub-repair/1 2019-12-21T16:00:17.321 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local 'ceph_args=--fsid=973e3fc1-e63e-4bde-9db6-84c68fd5e93e --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2019-12-21T16:00:17.321 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2019-12-21T16:00:17.321 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-journal-size=100' 2019-12-21T16:00:17.321 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2019-12-21T16:00:17.321 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2019-12-21T16:00:17.322 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2019-12-21T16:00:17.322 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --chdir=' 2019-12-21T16:00:17.322 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+= 2019-12-21T16:00:17.322 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2019-12-21T16:00:17.322 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: get_asok_path 2019-12-21T16:00:17.322 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2019-12-21T16:00:17.322 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2019-12-21T16:00:17.323 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2019-12-21T16:00:17.323 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T16:00:17.323 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.119575 2019-12-21T16:00:17.323 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.119575/$cluster-$name.asok' 2019-12-21T16:00:17.323 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.119575/$cluster-$name.asok' 2019-12-21T16:00:17.323 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --debug-osd=20' 2019-12-21T16:00:17.324 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2019-12-21T16:00:17.324 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2019-12-21T16:00:17.324 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2019-12-21T16:00:17.324 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2019-12-21T16:00:17.324 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2019-12-21T16:00:17.324 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' ' 2019-12-21T16:00:17.324 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2019-12-21T16:00:17.325 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: mkdir -p td/osd-scrub-repair/1 2019-12-21T16:00:17.325 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: echo start osd.1 2019-12-21T16:00:17.325 INFO:tasks.workunit.client.0.smithi172.stderr:start osd.1 2019-12-21T16:00:17.325 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph-osd -i 1 --fsid=973e3fc1-e63e-4bde-9db6-84c68fd5e93e --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.119575/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2019-12-21T16:00:17.325 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: cat td/osd-scrub-repair/1/whoami 2019-12-21T16:00:17.326 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: '[' 1 = 1 ']' 2019-12-21T16:00:17.326 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph osd dump --format=json 2019-12-21T16:00:17.327 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: jq '.flags_set[]' 2019-12-21T16:00:17.327 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: grep -q '"noup"' 2019-12-21T16:00:17.340 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:00:17.338+0000 7f5ef96e4ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T16:00:17.342 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:00:17.340+0000 7f5ef96e4ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T16:00:17.345 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:00:17.343+0000 7f5ef96e4ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T16:00:17.553 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: wait_for_osd up 1 2019-12-21T16:00:17.554 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:914: wait_for_osd: local state=up 2019-12-21T16:00:17.554 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local id=1 2019-12-21T16:00:17.554 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: status=1 2019-12-21T16:00:17.554 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i=0 )) 2019-12-21T16:00:17.554 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i < 300 )) 2019-12-21T16:00:17.554 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: echo 0 2019-12-21T16:00:17.554 INFO:tasks.workunit.client.0.smithi172.stderr:0 2019-12-21T16:00:17.555 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: ceph osd dump 2019-12-21T16:00:17.555 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: grep 'osd.1 up' 2019-12-21T16:00:17.779 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: sleep 1 2019-12-21T16:00:17.861 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:00:17.859+0000 7f5ef96e4ec0 -1 Falling back to public interface 2019-12-21T16:00:18.699 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:00:18.698+0000 7f5ef96e4ec0 -1 osd.1 37 log_to_monitors {default=true} 2019-12-21T16:00:18.782 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i++ )) 2019-12-21T16:00:18.782 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i < 300 )) 2019-12-21T16:00:18.782 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: echo 1 2019-12-21T16:00:18.783 INFO:tasks.workunit.client.0.smithi172.stderr:1 2019-12-21T16:00:18.783 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: ceph osd dump 2019-12-21T16:00:18.783 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: grep 'osd.1 up' 2019-12-21T16:00:19.095 INFO:tasks.workunit.client.0.smithi172.stderr:osd.1 up in weight 1 up_from 40 up_thru 40 down_at 38 last_clean_interval [32,37) [v2:127.0.0.1:6810/133888,v1:127.0.0.1:6811/133888] [v2:127.0.0.1:6812/133888,v1:127.0.0.1:6813/133888] exists,up 9bb8026b-4cfb-486b-aa07-b00acd8d49cd 2019-12-21T16:00:19.095 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=0 2019-12-21T16:00:19.096 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: break 2019-12-21T16:00:19.096 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: return 0 2019-12-21T16:00:19.096 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1239: objectstore_tool: wait_for_clean 2019-12-21T16:00:19.096 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1548: wait_for_clean: local cmd= 2019-12-21T16:00:19.096 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1549: wait_for_clean: local num_active_clean=-1 2019-12-21T16:00:19.096 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1550: wait_for_clean: local cur_active_clean 2019-12-21T16:00:19.096 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1551: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2019-12-21T16:00:19.097 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1551: wait_for_clean: get_timeout_delays 90 .1 2019-12-21T16:00:19.097 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1494: get_timeout_delays: shopt -q -o xtrace 2019-12-21T16:00:19.097 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1494: get_timeout_delays: echo true 2019-12-21T16:00:19.097 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1494: get_timeout_delays: local trace=true 2019-12-21T16:00:19.097 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1495: get_timeout_delays: true 2019-12-21T16:00:19.097 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1495: get_timeout_delays: shopt -u -o xtrace 2019-12-21T16:00:19.187 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1551: wait_for_clean: local -a delays 2019-12-21T16:00:19.187 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1552: wait_for_clean: local -i loop=0 2019-12-21T16:00:19.187 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1554: wait_for_clean: flush_pg_stats 2019-12-21T16:00:19.188 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1977: flush_pg_stats: local timeout=300 2019-12-21T16:00:19.188 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1979: flush_pg_stats: ceph osd ls 2019-12-21T16:00:19.448 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1979: flush_pg_stats: ids='0 2019-12-21T16:00:19.448 INFO:tasks.workunit.client.0.smithi172.stderr:1 2019-12-21T16:00:19.449 INFO:tasks.workunit.client.0.smithi172.stderr:2' 2019-12-21T16:00:19.449 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1980: flush_pg_stats: seqs= 2019-12-21T16:00:19.449 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: flush_pg_stats: for osd in $ids 2019-12-21T16:00:19.449 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2019-12-21T16:00:19.560 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: seq=154618822660 2019-12-21T16:00:19.560 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: flush_pg_stats: seqs=' 0-154618822660' 2019-12-21T16:00:19.560 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: flush_pg_stats: for osd in $ids 2019-12-21T16:00:19.560 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2019-12-21T16:00:19.671 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: seq=171798691842 2019-12-21T16:00:19.671 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: flush_pg_stats: seqs=' 0-154618822660 1-171798691842' 2019-12-21T16:00:19.671 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: flush_pg_stats: for osd in $ids 2019-12-21T16:00:19.671 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2019-12-21T16:00:19.781 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: seq=64424509462 2019-12-21T16:00:19.781 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: flush_pg_stats: seqs=' 0-154618822660 1-171798691842 2-64424509462' 2019-12-21T16:00:19.782 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: flush_pg_stats: for s in $seqs 2019-12-21T16:00:19.782 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: echo 0-154618822660 2019-12-21T16:00:19.782 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: cut -d - -f 1 2019-12-21T16:00:19.783 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: osd=0 2019-12-21T16:00:19.783 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: echo 0-154618822660 2019-12-21T16:00:19.783 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: cut -d - -f 2 2019-12-21T16:00:19.785 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: seq=154618822660 2019-12-21T16:00:19.785 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1989: flush_pg_stats: echo 'waiting osd.0 seq 154618822660' 2019-12-21T16:00:19.785 INFO:tasks.workunit.client.0.smithi172.stderr:waiting osd.0 seq 154618822660 2019-12-21T16:00:19.785 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 0 2019-12-21T16:00:20.011 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 154618822659 -lt 154618822660 2019-12-21T16:00:20.011 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1991: flush_pg_stats: sleep 1 2019-12-21T16:00:21.012 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1992: flush_pg_stats: '[' 300 -eq 0 ']' 2019-12-21T16:00:21.012 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 0 2019-12-21T16:00:21.245 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 154618822660 -lt 154618822660 2019-12-21T16:00:21.245 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: flush_pg_stats: for s in $seqs 2019-12-21T16:00:21.246 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: echo 1-171798691842 2019-12-21T16:00:21.246 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: cut -d - -f 1 2019-12-21T16:00:21.246 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: osd=1 2019-12-21T16:00:21.247 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: echo 1-171798691842 2019-12-21T16:00:21.247 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: cut -d - -f 2 2019-12-21T16:00:21.248 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: seq=171798691842 2019-12-21T16:00:21.248 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1989: flush_pg_stats: echo 'waiting osd.1 seq 171798691842' 2019-12-21T16:00:21.248 INFO:tasks.workunit.client.0.smithi172.stderr:waiting osd.1 seq 171798691842 2019-12-21T16:00:21.248 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 1 2019-12-21T16:00:21.475 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 171798691842 -lt 171798691842 2019-12-21T16:00:21.475 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: flush_pg_stats: for s in $seqs 2019-12-21T16:00:21.476 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: echo 2-64424509462 2019-12-21T16:00:21.476 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: cut -d - -f 1 2019-12-21T16:00:21.476 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: osd=2 2019-12-21T16:00:21.477 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: echo 2-64424509462 2019-12-21T16:00:21.477 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: cut -d - -f 2 2019-12-21T16:00:21.477 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: seq=64424509462 2019-12-21T16:00:21.478 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1989: flush_pg_stats: echo 'waiting osd.2 seq 64424509462' 2019-12-21T16:00:21.478 INFO:tasks.workunit.client.0.smithi172.stderr:waiting osd.2 seq 64424509462 2019-12-21T16:00:21.478 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 2 2019-12-21T16:00:21.707 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 64424509462 -lt 64424509462 2019-12-21T16:00:21.708 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1555: wait_for_clean: get_num_pgs 2019-12-21T16:00:21.708 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: ceph --format json status 2019-12-21T16:00:21.708 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: jq .pgmap.num_pgs 2019-12-21T16:00:21.978 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1555: wait_for_clean: test 1 == 0 2019-12-21T16:00:21.978 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1559: wait_for_clean: true 2019-12-21T16:00:21.978 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1563: wait_for_clean: get_num_active_clean 2019-12-21T16:00:21.978 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: get_num_active_clean: local expression 2019-12-21T16:00:21.978 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2019-12-21T16:00:21.979 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("stale") | not)' 2019-12-21T16:00:21.979 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: ceph --format json pg dump pgs 2019-12-21T16:00:21.979 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2019-12-21T16:00:22.199 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1563: wait_for_clean: cur_active_clean=1 2019-12-21T16:00:22.199 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1564: wait_for_clean: get_num_pgs 2019-12-21T16:00:22.200 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: ceph --format json status 2019-12-21T16:00:22.200 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: jq .pgmap.num_pgs 2019-12-21T16:00:22.470 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1564: wait_for_clean: test 1 = 1 2019-12-21T16:00:22.470 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1564: wait_for_clean: break 2019-12-21T16:00:22.470 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: return 0 2019-12-21T16:00:22.470 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:494: TEST_auto_repair_bluestore_failed: repair 1.0 2019-12-21T16:00:22.470 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: repair: local pgid=1.0 2019-12-21T16:00:22.470 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1665: repair: get_last_scrub_stamp 1.0 2019-12-21T16:00:22.471 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.0 2019-12-21T16:00:22.471 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T16:00:22.471 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T16:00:22.471 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2019-12-21T16:00:22.691 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1665: repair: local last_scrub=2019-12-21T15:59:58.064980+0000 2019-12-21T16:00:22.692 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1666: repair: ceph pg repair 1.0 2019-12-21T16:00:22.907 INFO:tasks.workunit.client.0.smithi172.stderr:instructing pg 1.0 on osd.1 to repair 2019-12-21T16:00:22.916 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: repair: wait_for_scrub 1.0 2019-12-21T15:59:58.064980+0000 2019-12-21T16:00:22.916 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1789: wait_for_scrub: local pgid=1.0 2019-12-21T16:00:22.916 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1790: wait_for_scrub: local last_scrub=2019-12-21T15:59:58.064980+0000 2019-12-21T16:00:22.916 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1791: wait_for_scrub: local sname=last_scrub_stamp 2019-12-21T16:00:22.916 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i=0 )) 2019-12-21T16:00:22.917 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T16:00:22.917 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2019-12-21T16:00:22.917 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.0 2019-12-21T16:00:22.917 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T16:00:22.917 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T16:00:22.917 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2019-12-21T16:00:23.138 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:59:58.064980+0000 '>' 2019-12-21T15:59:58.064980+0000 2019-12-21T16:00:23.138 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T16:00:24.139 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T16:00:24.139 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T16:00:24.140 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2019-12-21T16:00:24.140 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.0 2019-12-21T16:00:24.140 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T16:00:24.140 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T16:00:24.141 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2019-12-21T16:00:24.373 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:59:58.064980+0000 '>' 2019-12-21T15:59:58.064980+0000 2019-12-21T16:00:24.374 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T16:00:25.374 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T16:00:25.375 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T16:00:25.375 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2019-12-21T16:00:25.376 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.0 2019-12-21T16:00:25.376 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T16:00:25.376 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T16:00:25.376 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2019-12-21T16:00:25.609 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:59:58.064980+0000 '>' 2019-12-21T15:59:58.064980+0000 2019-12-21T16:00:25.609 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T16:00:26.611 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T16:00:26.611 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T16:00:26.611 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2019-12-21T16:00:26.612 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.0 2019-12-21T16:00:26.612 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T16:00:26.612 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T16:00:26.612 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2019-12-21T16:00:26.845 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:59:58.064980+0000 '>' 2019-12-21T15:59:58.064980+0000 2019-12-21T16:00:26.845 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T16:00:27.849 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T16:00:27.849 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T16:00:27.850 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2019-12-21T16:00:27.850 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.0 2019-12-21T16:00:27.850 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T16:00:27.850 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T16:00:27.851 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2019-12-21T16:00:28.079 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:59:58.064980+0000 '>' 2019-12-21T15:59:58.064980+0000 2019-12-21T16:00:28.079 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T16:00:29.080 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T16:00:29.080 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T16:00:29.081 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2019-12-21T16:00:29.081 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.0 2019-12-21T16:00:29.082 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T16:00:29.082 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T16:00:29.082 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2019-12-21T16:00:29.317 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T15:59:58.064980+0000 '>' 2019-12-21T15:59:58.064980+0000 2019-12-21T16:00:29.317 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T16:00:30.318 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T16:00:30.319 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T16:00:30.319 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2019-12-21T16:00:30.319 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.0 2019-12-21T16:00:30.319 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T16:00:30.320 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T16:00:30.320 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2019-12-21T16:00:30.555 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T16:00:26.680805+0000 '>' 2019-12-21T15:59:58.064980+0000 2019-12-21T16:00:30.555 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1795: wait_for_scrub: return 0 2019-12-21T16:00:30.555 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:495: TEST_auto_repair_bluestore_failed: sleep 2 2019-12-21T16:00:32.555 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:497: TEST_auto_repair_bluestore_failed: ceph pg dump pgs 2019-12-21T16:00:32.776 INFO:tasks.workunit.client.0.smithi172.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2019-12-21T16:00:32.776 INFO:tasks.workunit.client.0.smithi172.stdout:1.0 9 0 0 0 0 63 0 0 10 10 active+clean 2019-12-21T16:00:26.680904+0000 21'10 41:118 [1,0] 1 [1,0] 1 21'10 2019-12-21T16:00:26.680805+0000 21'10 2019-12-21T16:00:26.680805+0000 0 2019-12-21T16:00:32.776 INFO:tasks.workunit.client.0.smithi172.stdout: 2019-12-21T16:00:32.777 INFO:tasks.workunit.client.0.smithi172.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilisation. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2019-12-21T16:00:32.778 INFO:tasks.workunit.client.0.smithi172.stderr:dumped pgs 2019-12-21T16:00:32.786 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:498: TEST_auto_repair_bluestore_failed: ceph pg dump pgs 2019-12-21T16:00:32.786 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:498: TEST_auto_repair_bluestore_failed: pgid 2019-12-21T16:00:32.787 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh: line 498: pgid: command not found 2019-12-21T16:00:32.787 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:498: TEST_auto_repair_bluestore_failed: grep -q '^.* active+clean ' 2019-12-21T16:00:33.001 INFO:tasks.workunit.client.0.smithi172.stderr:dumped pgs 2019-12-21T16:00:33.009 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:499: TEST_auto_repair_bluestore_failed: grep scrub_finish td/osd-scrub-repair/osd.1.log 2019-12-21T16:00:33.011 INFO:tasks.workunit.client.0.smithi172.stdout:2019-12-21T15:59:57.837+0000 7fe9ef9bb700 20 osd.1 pg_epoch: 33 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=32/33 n=10 ec=17/17 lis/c=32/32 les/c/f=33/33/0 sis=32) [1,0] r=0 lpr=32 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+repair AUTO_REPAIR] scrub_finish 2019-12-21T16:00:33.012 INFO:tasks.workunit.client.0.smithi172.stdout:2019-12-21T15:59:57.837+0000 7fe9ef9bb700 20 osd.1 pg_epoch: 33 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=32/33 n=10 ec=17/17 lis/c=32/32 les/c/f=33/33/0 sis=32) [1,0] r=0 lpr=32 crt=21'10 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair AUTO_REPAIR mbc={255={(1+0)=1}}] scrub_finish Set scrub_after_recovery 2019-12-21T16:00:33.012 INFO:tasks.workunit.client.0.smithi172.stdout:2019-12-21T15:59:58.063+0000 7fe9ef9bb700 20 osd.1 pg_epoch: 33 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=32/33 n=10 ec=17/17 lis/c=32/32 les/c/f=33/33/0 sis=32) [1,0] r=0 lpr=32 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent CHECK_REPAIR] scrub_finish 2019-12-21T16:00:33.012 INFO:tasks.workunit.client.0.smithi172.stdout:2019-12-21T15:59:58.063+0000 7fe9ef9bb700 10 osd.1 pg_epoch: 33 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=32/33 n=10 ec=17/17 lis/c=32/32 les/c/f=33/33/0 sis=32) [1,0] r=0 lpr=32 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent+failed_repair] scrub_finish 3 error(s) still present after re-scrub 2019-12-21T16:00:33.012 INFO:tasks.workunit.client.0.smithi172.stdout:2019-12-21T16:00:26.679+0000 7f5ed7894700 20 osd.1 pg_epoch: 41 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=40/41 n=10 ec=17/17 lis/c=40/40 les/c/f=41/41/0 sis=40) [1,0] r=0 lpr=40 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent+repair] scrub_finish 2019-12-21T16:00:33.012 INFO:tasks.workunit.client.0.smithi172.stdout:2019-12-21T16:00:26.679+0000 7f5ed7894700 20 osd.1 pg_epoch: 41 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=40/41 n=9 ec=17/17 lis/c=40/40 les/c/f=41/41/0 sis=40) [1,0] r=0 lpr=40 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+repair] scrub_finish All may be fixed 2019-12-21T16:00:33.012 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:502: TEST_auto_repair_bluestore_failed: teardown td/osd-scrub-repair 2019-12-21T16:00:33.013 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2019-12-21T16:00:33.013 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2019-12-21T16:00:33.013 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2019-12-21T16:00:33.013 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2019-12-21T16:00:33.013 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2019-12-21T16:00:33.013 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2019-12-21T16:00:33.013 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2019-12-21T16:00:33.014 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2019-12-21T16:00:33.119 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2019-12-21T16:00:33.119 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2019-12-21T16:00:33.119 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2019-12-21T16:00:33.120 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2019-12-21T16:00:33.121 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2019-12-21T16:00:33.121 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2019-12-21T16:00:33.121 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2019-12-21T16:00:33.123 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local 'pattern=|/usr/lib/systemd/systemd-coredump %P %u %g %s %t %c %h %e' 2019-12-21T16:00:33.123 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' '|' = '|' ']' 2019-12-21T16:00:33.123 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:177: teardown: pattern= 2019-12-21T16:00:33.123 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2019-12-21T16:00:33.123 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname '' 2019-12-21T16:00:33.124 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls . 2019-12-21T16:00:33.124 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2019-12-21T16:00:33.125 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2019-12-21T16:00:33.133 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2019-12-21T16:00:33.133 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T16:00:33.133 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.119575 2019-12-21T16:00:33.133 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.119575 2019-12-21T16:00:33.135 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2019-12-21T16:00:33.135 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2019-12-21T16:00:33.135 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:62: run: for func in $funcs 2019-12-21T16:00:33.136 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: TEST_auto_repair_bluestore_failed_norecov td/osd-scrub-repair 2019-12-21T16:00:33.136 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:506: TEST_auto_repair_bluestore_failed_norecov: local dir=td/osd-scrub-repair 2019-12-21T16:00:33.136 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:507: TEST_auto_repair_bluestore_failed_norecov: local poolname=testpool 2019-12-21T16:00:33.136 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:510: TEST_auto_repair_bluestore_failed_norecov: setup td/osd-scrub-repair 2019-12-21T16:00:33.136 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2019-12-21T16:00:33.136 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2019-12-21T16:00:33.136 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2019-12-21T16:00:33.137 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2019-12-21T16:00:33.137 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2019-12-21T16:00:33.137 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2019-12-21T16:00:33.137 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2019-12-21T16:00:33.137 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2019-12-21T16:00:33.137 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2019-12-21T16:00:33.138 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2019-12-21T16:00:33.139 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2019-12-21T16:00:33.139 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2019-12-21T16:00:33.139 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2019-12-21T16:00:33.139 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2019-12-21T16:00:33.139 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2019-12-21T16:00:33.139 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2019-12-21T16:00:33.140 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2019-12-21T16:00:33.140 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local 'pattern=|/usr/lib/systemd/systemd-coredump %P %u %g %s %t %c %h %e' 2019-12-21T16:00:33.141 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' '|' = '|' ']' 2019-12-21T16:00:33.141 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:177: teardown: pattern= 2019-12-21T16:00:33.141 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2019-12-21T16:00:33.141 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname '' 2019-12-21T16:00:33.141 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls . 2019-12-21T16:00:33.142 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2019-12-21T16:00:33.142 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2019-12-21T16:00:33.143 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2019-12-21T16:00:33.143 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T16:00:33.143 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.119575 2019-12-21T16:00:33.143 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.119575 2019-12-21T16:00:33.144 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2019-12-21T16:00:33.144 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2019-12-21T16:00:33.144 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2019-12-21T16:00:33.145 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2019-12-21T16:00:33.145 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T16:00:33.145 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.119575 2019-12-21T16:00:33.145 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.119575 2019-12-21T16:00:33.147 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2019-12-21T16:00:33.147 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2019-12-21T16:00:33.147 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2019-12-21T16:00:33.147 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2019-12-21T16:00:33.147 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:511: TEST_auto_repair_bluestore_failed_norecov: run_mon td/osd-scrub-repair a 2019-12-21T16:00:33.148 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2019-12-21T16:00:33.148 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2019-12-21T16:00:33.148 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2019-12-21T16:00:33.148 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2019-12-21T16:00:33.148 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2019-12-21T16:00:33.148 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2019-12-21T16:00:33.403 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2019-12-21T16:00:33.403 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2019-12-21T16:00:33.404 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2019-12-21T16:00:33.404 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2019-12-21T16:00:33.404 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T16:00:33.404 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.119575 2019-12-21T16:00:33.404 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.119575/$cluster-$name.asok' 2019-12-21T16:00:33.404 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.119575/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 2019-12-21T16:00:33.578 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: cat 2019-12-21T16:00:33.580 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: get_config mon a fsid 2019-12-21T16:00:33.580 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: get_config: local daemon=mon 2019-12-21T16:00:33.580 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: get_config: local id=a 2019-12-21T16:00:33.580 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local config=fsid 2019-12-21T16:00:33.580 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: jq -r .fsid 2019-12-21T16:00:33.581 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: get_asok_path mon.a 2019-12-21T16:00:33.581 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2019-12-21T16:00:33.581 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2019-12-21T16:00:33.581 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T16:00:33.581 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T16:00:33.581 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.119575 2019-12-21T16:00:33.581 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.119575/ceph-mon.a.asok 2019-12-21T16:00:33.582 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: CEPH_ARGS= 2019-12-21T16:00:33.582 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: ceph --format json daemon /tmp/ceph-asok.119575/ceph-mon.a.asok config get fsid 2019-12-21T16:00:33.669 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: get_config mon a mon_host 2019-12-21T16:00:33.669 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: get_config: local daemon=mon 2019-12-21T16:00:33.670 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: get_config: local id=a 2019-12-21T16:00:33.670 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local config=mon_host 2019-12-21T16:00:33.670 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: jq -r .mon_host 2019-12-21T16:00:33.670 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: get_asok_path mon.a 2019-12-21T16:00:33.670 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2019-12-21T16:00:33.670 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2019-12-21T16:00:33.670 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T16:00:33.671 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T16:00:33.671 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.119575 2019-12-21T16:00:33.671 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.119575/ceph-mon.a.asok 2019-12-21T16:00:33.671 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: CEPH_ARGS= 2019-12-21T16:00:33.671 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: ceph --format json daemon /tmp/ceph-asok.119575/ceph-mon.a.asok config get mon_host 2019-12-21T16:00:33.757 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:512: TEST_auto_repair_bluestore_failed_norecov: run_mgr td/osd-scrub-repair x 2019-12-21T16:00:33.757 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:551: run_mgr: local dir=td/osd-scrub-repair 2019-12-21T16:00:33.758 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: shift 2019-12-21T16:00:33.758 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local id=x 2019-12-21T16:00:33.758 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2019-12-21T16:00:33.758 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local data=td/osd-scrub-repair/x 2019-12-21T16:00:33.758 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:558: run_mgr: get_asok_path 2019-12-21T16:00:33.758 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2019-12-21T16:00:33.759 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2019-12-21T16:00:33.759 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2019-12-21T16:00:33.759 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T16:00:33.759 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.119575 2019-12-21T16:00:33.759 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.119575/$cluster-$name.asok' 2019-12-21T16:00:33.760 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:558: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2019-12-21T16:00:33.760 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:558: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.119575/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2019-12-21T16:00:34.381 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:515: TEST_auto_repair_bluestore_failed_norecov: local 'ceph_osd_args=--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2019-12-21T16:00:34.382 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:516: TEST_auto_repair_bluestore_failed_norecov: seq 0 2 2019-12-21T16:00:34.385 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:516: TEST_auto_repair_bluestore_failed_norecov: for id in $(seq 0 2) 2019-12-21T16:00:34.385 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:517: TEST_auto_repair_bluestore_failed_norecov: run_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2019-12-21T16:00:34.386 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local dir=td/osd-scrub-repair 2019-12-21T16:00:34.386 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: shift 2019-12-21T16:00:34.386 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local id=0 2019-12-21T16:00:34.386 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2019-12-21T16:00:34.386 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local osd_data=td/osd-scrub-repair/0 2019-12-21T16:00:34.386 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local 'ceph_args=--fsid=973e3fc1-e63e-4bde-9db6-84c68fd5e93e --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2019-12-21T16:00:34.386 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2019-12-21T16:00:34.387 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-journal-size=100' 2019-12-21T16:00:34.387 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2019-12-21T16:00:34.387 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2019-12-21T16:00:34.387 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2019-12-21T16:00:34.387 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --chdir=' 2019-12-21T16:00:34.387 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+= 2019-12-21T16:00:34.387 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2019-12-21T16:00:34.388 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: get_asok_path 2019-12-21T16:00:34.388 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2019-12-21T16:00:34.388 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2019-12-21T16:00:34.388 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2019-12-21T16:00:34.388 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T16:00:34.388 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.119575 2019-12-21T16:00:34.388 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.119575/$cluster-$name.asok' 2019-12-21T16:00:34.389 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.119575/$cluster-$name.asok' 2019-12-21T16:00:34.389 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --debug-osd=20' 2019-12-21T16:00:34.389 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-ms=1' 2019-12-21T16:00:34.389 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-monc=20' 2019-12-21T16:00:34.389 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2019-12-21T16:00:34.389 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2019-12-21T16:00:34.389 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2019-12-21T16:00:34.390 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2019-12-21T16:00:34.390 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2019-12-21T16:00:34.390 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' ' 2019-12-21T16:00:34.390 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2019-12-21T16:00:34.390 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: mkdir -p td/osd-scrub-repair/0 2019-12-21T16:00:34.391 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: uuidgen 2019-12-21T16:00:34.391 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: local uuid=cfa50606-889a-4721-96df-45a2bf205408 2019-12-21T16:00:34.391 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: echo 'add osd0 cfa50606-889a-4721-96df-45a2bf205408' 2019-12-21T16:00:34.391 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: ceph-authtool --gen-print-key 2019-12-21T16:00:34.392 INFO:tasks.workunit.client.0.smithi172.stdout:add osd0 cfa50606-889a-4721-96df-45a2bf205408 2019-12-21T16:00:34.400 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: OSD_SECRET=AQCiQf5dPmfHFxAAiEzdbzOi2hRY04PQwwPDuw== 2019-12-21T16:00:34.401 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo '{"cephx_secret": "AQCiQf5dPmfHFxAAiEzdbzOi2hRY04PQwwPDuw=="}' 2019-12-21T16:00:34.401 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph osd new cfa50606-889a-4721-96df-45a2bf205408 -i td/osd-scrub-repair/0/new.json 2019-12-21T16:00:34.644 INFO:tasks.workunit.client.0.smithi172.stdout:0 2019-12-21T16:00:34.654 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: rm td/osd-scrub-repair/0/new.json 2019-12-21T16:00:34.656 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-osd -i 0 --fsid=973e3fc1-e63e-4bde-9db6-84c68fd5e93e --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.119575/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQCiQf5dPmfHFxAAiEzdbzOi2hRY04PQwwPDuw== --osd-uuid cfa50606-889a-4721-96df-45a2bf205408 2019-12-21T16:00:34.678 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:00:34.675+0000 7fca0debcec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T16:00:34.680 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:00:34.677+0000 7fca0debcec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T16:00:34.681 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:00:34.678+0000 7fca0debcec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T16:00:34.682 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:00:34.679+0000 7fca0debcec0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2019-12-21T16:00:34.682 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:00:34.679+0000 7fca0debcec0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2019-12-21T16:00:34.682 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:00:34.679+0000 7fca0debcec0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2019-12-21T16:00:34.719 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:00:34.716+0000 7fca0debcec0 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2019-12-21T16:00:37.759 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2019-12-21T16:00:37.760 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: cat 2019-12-21T16:00:37.760 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: echo adding osd0 key to auth repository 2019-12-21T16:00:37.761 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2019-12-21T16:00:37.761 INFO:tasks.workunit.client.0.smithi172.stdout:adding osd0 key to auth repository 2019-12-21T16:00:38.039 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo start osd.0 2019-12-21T16:00:38.039 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph-osd -i 0 --fsid=973e3fc1-e63e-4bde-9db6-84c68fd5e93e --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.119575/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2019-12-21T16:00:38.039 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: jq '.flags_set[]' 2019-12-21T16:00:38.039 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph osd dump --format=json 2019-12-21T16:00:38.039 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: grep -q '"noup"' 2019-12-21T16:00:38.040 INFO:tasks.workunit.client.0.smithi172.stdout:start osd.0 2019-12-21T16:00:38.061 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:00:38.058+0000 7fc6881ccec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T16:00:38.070 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:00:38.067+0000 7fc6881ccec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T16:00:38.072 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:00:38.070+0000 7fc6881ccec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T16:00:38.270 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: wait_for_osd up 0 2019-12-21T16:00:38.271 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:914: wait_for_osd: local state=up 2019-12-21T16:00:38.271 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local id=0 2019-12-21T16:00:38.271 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: status=1 2019-12-21T16:00:38.271 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i=0 )) 2019-12-21T16:00:38.271 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i < 300 )) 2019-12-21T16:00:38.271 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: echo 0 2019-12-21T16:00:38.271 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: grep 'osd.0 up' 2019-12-21T16:00:38.272 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: ceph osd dump 2019-12-21T16:00:38.272 INFO:tasks.workunit.client.0.smithi172.stdout:0 2019-12-21T16:00:38.496 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: sleep 1 2019-12-21T16:00:38.592 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:00:38.590+0000 7fc6881ccec0 -1 Falling back to public interface 2019-12-21T16:00:39.327 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:00:39.324+0000 7fc6881ccec0 -1 osd.0 0 log_to_monitors {default=true} 2019-12-21T16:00:39.497 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i++ )) 2019-12-21T16:00:39.498 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i < 300 )) 2019-12-21T16:00:39.498 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: echo 1 2019-12-21T16:00:39.498 INFO:tasks.workunit.client.0.smithi172.stdout:1 2019-12-21T16:00:39.498 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: grep 'osd.0 up' 2019-12-21T16:00:39.499 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: ceph osd dump 2019-12-21T16:00:39.799 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: sleep 1 2019-12-21T16:00:40.801 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i++ )) 2019-12-21T16:00:40.801 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i < 300 )) 2019-12-21T16:00:40.801 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: echo 2 2019-12-21T16:00:40.801 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: ceph osd dump 2019-12-21T16:00:40.801 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: grep 'osd.0 up' 2019-12-21T16:00:40.802 INFO:tasks.workunit.client.0.smithi172.stdout:2 2019-12-21T16:00:41.034 INFO:tasks.workunit.client.0.smithi172.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/135168,v1:127.0.0.1:6803/135168] [v2:127.0.0.1:6804/135168,v1:127.0.0.1:6805/135168] exists,up cfa50606-889a-4721-96df-45a2bf205408 2019-12-21T16:00:41.035 INFO:tasks.workunit.client.0.smithi172.stdout:add osd1 3a5b611e-245f-49af-9fac-3bc6b41e6466 2019-12-21T16:00:41.035 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=0 2019-12-21T16:00:41.035 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: break 2019-12-21T16:00:41.035 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: return 0 2019-12-21T16:00:41.035 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:516: TEST_auto_repair_bluestore_failed_norecov: for id in $(seq 0 2) 2019-12-21T16:00:41.035 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:517: TEST_auto_repair_bluestore_failed_norecov: run_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2019-12-21T16:00:41.036 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local dir=td/osd-scrub-repair 2019-12-21T16:00:41.036 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: shift 2019-12-21T16:00:41.036 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local id=1 2019-12-21T16:00:41.036 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2019-12-21T16:00:41.036 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local osd_data=td/osd-scrub-repair/1 2019-12-21T16:00:41.036 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local 'ceph_args=--fsid=973e3fc1-e63e-4bde-9db6-84c68fd5e93e --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2019-12-21T16:00:41.037 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2019-12-21T16:00:41.037 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-journal-size=100' 2019-12-21T16:00:41.037 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2019-12-21T16:00:41.037 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2019-12-21T16:00:41.037 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2019-12-21T16:00:41.037 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --chdir=' 2019-12-21T16:00:41.037 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+= 2019-12-21T16:00:41.038 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2019-12-21T16:00:41.038 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: get_asok_path 2019-12-21T16:00:41.038 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2019-12-21T16:00:41.038 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2019-12-21T16:00:41.038 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2019-12-21T16:00:41.038 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T16:00:41.038 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.119575 2019-12-21T16:00:41.039 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.119575/$cluster-$name.asok' 2019-12-21T16:00:41.039 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.119575/$cluster-$name.asok' 2019-12-21T16:00:41.039 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --debug-osd=20' 2019-12-21T16:00:41.039 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-ms=1' 2019-12-21T16:00:41.039 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-monc=20' 2019-12-21T16:00:41.039 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2019-12-21T16:00:41.039 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2019-12-21T16:00:41.040 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2019-12-21T16:00:41.040 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2019-12-21T16:00:41.040 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2019-12-21T16:00:41.040 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' ' 2019-12-21T16:00:41.040 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2019-12-21T16:00:41.040 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: mkdir -p td/osd-scrub-repair/1 2019-12-21T16:00:41.042 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: uuidgen 2019-12-21T16:00:41.042 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: local uuid=3a5b611e-245f-49af-9fac-3bc6b41e6466 2019-12-21T16:00:41.042 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: echo 'add osd1 3a5b611e-245f-49af-9fac-3bc6b41e6466' 2019-12-21T16:00:41.043 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: ceph-authtool --gen-print-key 2019-12-21T16:00:41.047 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: OSD_SECRET=AQCpQf5dPWS0AhAAnVtQyk5v5R+YmY9ZsWFh6A== 2019-12-21T16:00:41.047 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo '{"cephx_secret": "AQCpQf5dPWS0AhAAnVtQyk5v5R+YmY9ZsWFh6A=="}' 2019-12-21T16:00:41.047 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph osd new 3a5b611e-245f-49af-9fac-3bc6b41e6466 -i td/osd-scrub-repair/1/new.json 2019-12-21T16:00:41.359 INFO:tasks.workunit.client.0.smithi172.stdout:1 2019-12-21T16:00:41.369 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: rm td/osd-scrub-repair/1/new.json 2019-12-21T16:00:41.370 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-osd -i 1 --fsid=973e3fc1-e63e-4bde-9db6-84c68fd5e93e --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.119575/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQCpQf5dPWS0AhAAnVtQyk5v5R+YmY9ZsWFh6A== --osd-uuid 3a5b611e-245f-49af-9fac-3bc6b41e6466 2019-12-21T16:00:41.392 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:00:41.390+0000 7f400b65eec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T16:00:41.394 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:00:41.392+0000 7f400b65eec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T16:00:41.395 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:00:41.393+0000 7f400b65eec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T16:00:41.396 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:00:41.393+0000 7f400b65eec0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2019-12-21T16:00:41.396 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:00:41.393+0000 7f400b65eec0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2019-12-21T16:00:41.396 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:00:41.393+0000 7f400b65eec0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2019-12-21T16:00:41.450 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:00:41.448+0000 7f400b65eec0 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2019-12-21T16:00:44.534 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2019-12-21T16:00:44.535 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: cat 2019-12-21T16:00:44.536 INFO:tasks.workunit.client.0.smithi172.stdout:adding osd1 key to auth repository 2019-12-21T16:00:44.536 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: echo adding osd1 key to auth repository 2019-12-21T16:00:44.536 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2019-12-21T16:00:44.818 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo start osd.1 2019-12-21T16:00:44.818 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph-osd -i 1 --fsid=973e3fc1-e63e-4bde-9db6-84c68fd5e93e --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.119575/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2019-12-21T16:00:44.819 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph osd dump --format=json 2019-12-21T16:00:44.819 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: jq '.flags_set[]' 2019-12-21T16:00:44.819 INFO:tasks.workunit.client.0.smithi172.stdout:start osd.1 2019-12-21T16:00:44.819 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: grep -q '"noup"' 2019-12-21T16:00:44.839 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:00:44.837+0000 7f178f62fec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T16:00:44.841 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:00:44.838+0000 7f178f62fec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T16:00:44.843 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:00:44.840+0000 7f178f62fec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T16:00:45.047 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: wait_for_osd up 1 2019-12-21T16:00:45.047 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:914: wait_for_osd: local state=up 2019-12-21T16:00:45.047 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local id=1 2019-12-21T16:00:45.047 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: status=1 2019-12-21T16:00:45.047 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i=0 )) 2019-12-21T16:00:45.048 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i < 300 )) 2019-12-21T16:00:45.048 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: echo 0 2019-12-21T16:00:45.048 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: grep 'osd.1 up' 2019-12-21T16:00:45.048 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: ceph osd dump 2019-12-21T16:00:45.048 INFO:tasks.workunit.client.0.smithi172.stdout:0 2019-12-21T16:00:45.274 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: sleep 1 2019-12-21T16:00:45.358 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:00:45.357+0000 7f178f62fec0 -1 Falling back to public interface 2019-12-21T16:00:46.099 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:00:46.097+0000 7f178f62fec0 -1 osd.1 0 log_to_monitors {default=true} 2019-12-21T16:00:46.275 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i++ )) 2019-12-21T16:00:46.275 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i < 300 )) 2019-12-21T16:00:46.276 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: echo 1 2019-12-21T16:00:46.276 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: ceph osd dump 2019-12-21T16:00:46.276 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: grep 'osd.1 up' 2019-12-21T16:00:46.276 INFO:tasks.workunit.client.0.smithi172.stdout:1 2019-12-21T16:00:46.536 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: sleep 1 2019-12-21T16:00:47.537 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i++ )) 2019-12-21T16:00:47.537 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i < 300 )) 2019-12-21T16:00:47.538 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: echo 2 2019-12-21T16:00:47.538 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: ceph osd dump 2019-12-21T16:00:47.538 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: grep 'osd.1 up' 2019-12-21T16:00:47.538 INFO:tasks.workunit.client.0.smithi172.stdout:2 2019-12-21T16:00:47.768 INFO:tasks.workunit.client.0.smithi172.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/135545,v1:127.0.0.1:6811/135545] [v2:127.0.0.1:6812/135545,v1:127.0.0.1:6813/135545] exists,up 3a5b611e-245f-49af-9fac-3bc6b41e6466 2019-12-21T16:00:47.769 INFO:tasks.workunit.client.0.smithi172.stdout:add osd2 00287a12-5406-407a-b5d2-c8430e002e98 2019-12-21T16:00:47.769 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=0 2019-12-21T16:00:47.769 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: break 2019-12-21T16:00:47.769 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: return 0 2019-12-21T16:00:47.770 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:516: TEST_auto_repair_bluestore_failed_norecov: for id in $(seq 0 2) 2019-12-21T16:00:47.770 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:517: TEST_auto_repair_bluestore_failed_norecov: run_osd td/osd-scrub-repair 2 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2019-12-21T16:00:47.770 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local dir=td/osd-scrub-repair 2019-12-21T16:00:47.770 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: shift 2019-12-21T16:00:47.770 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local id=2 2019-12-21T16:00:47.770 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2019-12-21T16:00:47.770 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local osd_data=td/osd-scrub-repair/2 2019-12-21T16:00:47.771 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local 'ceph_args=--fsid=973e3fc1-e63e-4bde-9db6-84c68fd5e93e --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2019-12-21T16:00:47.771 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2019-12-21T16:00:47.771 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-journal-size=100' 2019-12-21T16:00:47.771 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2019-12-21T16:00:47.771 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2019-12-21T16:00:47.771 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2019-12-21T16:00:47.772 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --chdir=' 2019-12-21T16:00:47.772 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+= 2019-12-21T16:00:47.772 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2019-12-21T16:00:47.772 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: get_asok_path 2019-12-21T16:00:47.773 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2019-12-21T16:00:47.773 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2019-12-21T16:00:47.774 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2019-12-21T16:00:47.774 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T16:00:47.774 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.119575 2019-12-21T16:00:47.774 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.119575/$cluster-$name.asok' 2019-12-21T16:00:47.774 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.119575/$cluster-$name.asok' 2019-12-21T16:00:47.775 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --debug-osd=20' 2019-12-21T16:00:47.775 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-ms=1' 2019-12-21T16:00:47.775 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-monc=20' 2019-12-21T16:00:47.775 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2019-12-21T16:00:47.775 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2019-12-21T16:00:47.775 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2019-12-21T16:00:47.776 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2019-12-21T16:00:47.776 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2019-12-21T16:00:47.776 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' ' 2019-12-21T16:00:47.776 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2019-12-21T16:00:47.776 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: mkdir -p td/osd-scrub-repair/2 2019-12-21T16:00:47.776 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: uuidgen 2019-12-21T16:00:47.776 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: local uuid=00287a12-5406-407a-b5d2-c8430e002e98 2019-12-21T16:00:47.777 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: echo 'add osd2 00287a12-5406-407a-b5d2-c8430e002e98' 2019-12-21T16:00:47.777 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: ceph-authtool --gen-print-key 2019-12-21T16:00:47.780 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: OSD_SECRET=AQCvQf5d0+RrLhAA3GQaAYn2QqLzDy0guSb8aA== 2019-12-21T16:00:47.780 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo '{"cephx_secret": "AQCvQf5d0+RrLhAA3GQaAYn2QqLzDy0guSb8aA=="}' 2019-12-21T16:00:47.781 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph osd new 00287a12-5406-407a-b5d2-c8430e002e98 -i td/osd-scrub-repair/2/new.json 2019-12-21T16:00:48.091 INFO:tasks.workunit.client.0.smithi172.stdout:2 2019-12-21T16:00:48.101 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: rm td/osd-scrub-repair/2/new.json 2019-12-21T16:00:48.102 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-osd -i 2 --fsid=973e3fc1-e63e-4bde-9db6-84c68fd5e93e --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.119575/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQCvQf5d0+RrLhAA3GQaAYn2QqLzDy0guSb8aA== --osd-uuid 00287a12-5406-407a-b5d2-c8430e002e98 2019-12-21T16:00:48.124 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:00:48.122+0000 7f41e9d81ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T16:00:48.152 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:00:48.150+0000 7f41e9d81ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T16:00:48.153 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:00:48.151+0000 7f41e9d81ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T16:00:48.154 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:00:48.152+0000 7f41e9d81ec0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2019-12-21T16:00:48.154 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:00:48.152+0000 7f41e9d81ec0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2019-12-21T16:00:48.154 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:00:48.152+0000 7f41e9d81ec0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2019-12-21T16:00:48.215 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:00:48.213+0000 7f41e9d81ec0 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2019-12-21T16:00:51.436 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2019-12-21T16:00:51.436 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: cat 2019-12-21T16:00:51.437 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: echo adding osd2 key to auth repository 2019-12-21T16:00:51.437 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2019-12-21T16:00:51.437 INFO:tasks.workunit.client.0.smithi172.stdout:adding osd2 key to auth repository 2019-12-21T16:00:51.722 INFO:tasks.workunit.client.0.smithi172.stdout:start osd.2 2019-12-21T16:00:51.722 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo start osd.2 2019-12-21T16:00:51.723 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph-osd -i 2 --fsid=973e3fc1-e63e-4bde-9db6-84c68fd5e93e --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.119575/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2019-12-21T16:00:51.723 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph osd dump --format=json 2019-12-21T16:00:51.723 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: jq '.flags_set[]' 2019-12-21T16:00:51.723 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: grep -q '"noup"' 2019-12-21T16:00:51.743 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:00:51.742+0000 7ff820dd7ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T16:00:51.745 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:00:51.744+0000 7ff820dd7ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T16:00:51.747 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:00:51.745+0000 7ff820dd7ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T16:00:51.978 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: wait_for_osd up 2 2019-12-21T16:00:51.978 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:914: wait_for_osd: local state=up 2019-12-21T16:00:51.978 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local id=2 2019-12-21T16:00:51.978 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: status=1 2019-12-21T16:00:51.978 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i=0 )) 2019-12-21T16:00:51.979 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i < 300 )) 2019-12-21T16:00:51.979 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: echo 0 2019-12-21T16:00:51.979 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: ceph osd dump 2019-12-21T16:00:51.979 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: grep 'osd.2 up' 2019-12-21T16:00:51.979 INFO:tasks.workunit.client.0.smithi172.stdout:0 2019-12-21T16:00:52.205 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: sleep 1 2019-12-21T16:00:52.262 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:00:52.261+0000 7ff820dd7ec0 -1 Falling back to public interface 2019-12-21T16:00:53.006 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:00:53.004+0000 7ff820dd7ec0 -1 osd.2 0 log_to_monitors {default=true} 2019-12-21T16:00:53.206 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i++ )) 2019-12-21T16:00:53.207 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i < 300 )) 2019-12-21T16:00:53.207 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: echo 1 2019-12-21T16:00:53.207 INFO:tasks.workunit.client.0.smithi172.stdout:1 2019-12-21T16:00:53.207 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: ceph osd dump 2019-12-21T16:00:53.207 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: grep 'osd.2 up' 2019-12-21T16:00:53.461 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: sleep 1 2019-12-21T16:00:54.463 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i++ )) 2019-12-21T16:00:54.463 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i < 300 )) 2019-12-21T16:00:54.463 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: echo 2 2019-12-21T16:00:54.463 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: ceph osd dump 2019-12-21T16:00:54.463 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: grep 'osd.2 up' 2019-12-21T16:00:54.463 INFO:tasks.workunit.client.0.smithi172.stdout:2 2019-12-21T16:00:54.691 INFO:tasks.workunit.client.0.smithi172.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/135922,v1:127.0.0.1:6819/135922] [v2:127.0.0.1:6820/135922,v1:127.0.0.1:6821/135922] exists,up 00287a12-5406-407a-b5d2-c8430e002e98 2019-12-21T16:00:54.691 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=0 2019-12-21T16:00:54.691 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: break 2019-12-21T16:00:54.691 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: return 0 2019-12-21T16:00:54.691 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:520: TEST_auto_repair_bluestore_failed_norecov: create_pool testpool 1 1 2019-12-21T16:00:54.692 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:539: create_pool: ceph osd pool create testpool 1 1 2019-12-21T16:00:55.042 INFO:tasks.workunit.client.0.smithi172.stderr:pool 'testpool' created 2019-12-21T16:00:55.049 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: sleep 1 2019-12-21T16:00:56.051 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:521: TEST_auto_repair_bluestore_failed_norecov: ceph osd pool set testpool size 2 2019-12-21T16:00:56.494 INFO:tasks.workunit.client.0.smithi172.stderr:set pool 1 size to 2 2019-12-21T16:00:56.511 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:522: TEST_auto_repair_bluestore_failed_norecov: wait_for_clean 2019-12-21T16:00:56.511 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1548: wait_for_clean: local cmd= 2019-12-21T16:00:56.511 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1549: wait_for_clean: local num_active_clean=-1 2019-12-21T16:00:56.511 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1550: wait_for_clean: local cur_active_clean 2019-12-21T16:00:56.511 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1551: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2019-12-21T16:00:56.511 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1551: wait_for_clean: get_timeout_delays 90 .1 2019-12-21T16:00:56.512 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1494: get_timeout_delays: shopt -q -o xtrace 2019-12-21T16:00:56.512 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1494: get_timeout_delays: echo true 2019-12-21T16:00:56.512 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1494: get_timeout_delays: local trace=true 2019-12-21T16:00:56.512 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1495: get_timeout_delays: true 2019-12-21T16:00:56.512 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1495: get_timeout_delays: shopt -u -o xtrace 2019-12-21T16:00:56.596 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1551: wait_for_clean: local -a delays 2019-12-21T16:00:56.596 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1552: wait_for_clean: local -i loop=0 2019-12-21T16:00:56.596 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1554: wait_for_clean: flush_pg_stats 2019-12-21T16:00:56.596 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1977: flush_pg_stats: local timeout=300 2019-12-21T16:00:56.596 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1979: flush_pg_stats: ceph osd ls 2019-12-21T16:00:56.835 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1979: flush_pg_stats: ids='0 2019-12-21T16:00:56.836 INFO:tasks.workunit.client.0.smithi172.stderr:1 2019-12-21T16:00:56.836 INFO:tasks.workunit.client.0.smithi172.stderr:2' 2019-12-21T16:00:56.836 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1980: flush_pg_stats: seqs= 2019-12-21T16:00:56.836 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: flush_pg_stats: for osd in $ids 2019-12-21T16:00:56.836 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2019-12-21T16:00:56.946 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: seq=21474836485 2019-12-21T16:00:56.946 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: flush_pg_stats: seqs=' 0-21474836485' 2019-12-21T16:00:56.946 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: flush_pg_stats: for osd in $ids 2019-12-21T16:00:56.947 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2019-12-21T16:00:57.058 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: seq=42949672963 2019-12-21T16:00:57.058 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: flush_pg_stats: seqs=' 0-21474836485 1-42949672963' 2019-12-21T16:00:57.058 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: flush_pg_stats: for osd in $ids 2019-12-21T16:00:57.058 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2019-12-21T16:00:57.169 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: seq=64424509442 2019-12-21T16:00:57.169 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: flush_pg_stats: seqs=' 0-21474836485 1-42949672963 2-64424509442' 2019-12-21T16:00:57.169 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: flush_pg_stats: for s in $seqs 2019-12-21T16:00:57.170 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: echo 0-21474836485 2019-12-21T16:00:57.170 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: cut -d - -f 1 2019-12-21T16:00:57.170 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: osd=0 2019-12-21T16:00:57.171 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: echo 0-21474836485 2019-12-21T16:00:57.171 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: cut -d - -f 2 2019-12-21T16:00:57.172 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: seq=21474836485 2019-12-21T16:00:57.172 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1989: flush_pg_stats: echo 'waiting osd.0 seq 21474836485' 2019-12-21T16:00:57.172 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 0 2019-12-21T16:00:57.172 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.0 seq 21474836485 2019-12-21T16:00:57.399 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 21474836484 -lt 21474836485 2019-12-21T16:00:57.399 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1991: flush_pg_stats: sleep 1 2019-12-21T16:00:58.400 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1992: flush_pg_stats: '[' 300 -eq 0 ']' 2019-12-21T16:00:58.400 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 0 2019-12-21T16:00:58.636 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 21474836485 -lt 21474836485 2019-12-21T16:00:58.636 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: flush_pg_stats: for s in $seqs 2019-12-21T16:00:58.636 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: echo 1-42949672963 2019-12-21T16:00:58.637 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: cut -d - -f 1 2019-12-21T16:00:58.637 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: osd=1 2019-12-21T16:00:58.637 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: echo 1-42949672963 2019-12-21T16:00:58.638 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: cut -d - -f 2 2019-12-21T16:00:58.639 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: seq=42949672963 2019-12-21T16:00:58.639 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1989: flush_pg_stats: echo 'waiting osd.1 seq 42949672963' 2019-12-21T16:00:58.639 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 1 2019-12-21T16:00:58.639 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.1 seq 42949672963 2019-12-21T16:00:58.867 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 42949672964 -lt 42949672963 2019-12-21T16:00:58.868 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: flush_pg_stats: for s in $seqs 2019-12-21T16:00:58.868 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: echo 2-64424509442 2019-12-21T16:00:58.868 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: cut -d - -f 1 2019-12-21T16:00:58.868 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: osd=2 2019-12-21T16:00:58.869 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: echo 2-64424509442 2019-12-21T16:00:58.869 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: cut -d - -f 2 2019-12-21T16:00:58.870 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: seq=64424509442 2019-12-21T16:00:58.870 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1989: flush_pg_stats: echo 'waiting osd.2 seq 64424509442' 2019-12-21T16:00:58.870 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 2 2019-12-21T16:00:58.870 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.2 seq 64424509442 2019-12-21T16:00:59.097 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 64424509442 -lt 64424509442 2019-12-21T16:00:59.097 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1555: wait_for_clean: get_num_pgs 2019-12-21T16:00:59.097 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: ceph --format json status 2019-12-21T16:00:59.098 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: jq .pgmap.num_pgs 2019-12-21T16:00:59.366 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1555: wait_for_clean: test 1 == 0 2019-12-21T16:00:59.366 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1559: wait_for_clean: true 2019-12-21T16:00:59.366 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1563: wait_for_clean: get_num_active_clean 2019-12-21T16:00:59.367 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: get_num_active_clean: local expression 2019-12-21T16:00:59.367 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2019-12-21T16:00:59.367 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("stale") | not)' 2019-12-21T16:00:59.367 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: ceph --format json pg dump pgs 2019-12-21T16:00:59.367 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2019-12-21T16:00:59.587 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1563: wait_for_clean: cur_active_clean=1 2019-12-21T16:00:59.587 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1564: wait_for_clean: get_num_pgs 2019-12-21T16:00:59.588 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: ceph --format json status 2019-12-21T16:00:59.588 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: jq .pgmap.num_pgs 2019-12-21T16:00:59.858 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1564: wait_for_clean: test 1 = 1 2019-12-21T16:00:59.859 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1564: wait_for_clean: break 2019-12-21T16:00:59.859 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: return 0 2019-12-21T16:00:59.859 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:525: TEST_auto_repair_bluestore_failed_norecov: local payload=ABCDEF 2019-12-21T16:00:59.859 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:526: TEST_auto_repair_bluestore_failed_norecov: echo ABCDEF 2019-12-21T16:00:59.859 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:527: TEST_auto_repair_bluestore_failed_norecov: seq 1 10 2019-12-21T16:00:59.860 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:527: TEST_auto_repair_bluestore_failed_norecov: for i in $(seq 1 10) 2019-12-21T16:00:59.860 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:529: TEST_auto_repair_bluestore_failed_norecov: rados --pool testpool put obj1 td/osd-scrub-repair/ORIGINAL 2019-12-21T16:00:59.891 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:527: TEST_auto_repair_bluestore_failed_norecov: for i in $(seq 1 10) 2019-12-21T16:00:59.891 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:529: TEST_auto_repair_bluestore_failed_norecov: rados --pool testpool put obj2 td/osd-scrub-repair/ORIGINAL 2019-12-21T16:00:59.974 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:527: TEST_auto_repair_bluestore_failed_norecov: for i in $(seq 1 10) 2019-12-21T16:00:59.975 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:529: TEST_auto_repair_bluestore_failed_norecov: rados --pool testpool put obj3 td/osd-scrub-repair/ORIGINAL 2019-12-21T16:01:00.046 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:527: TEST_auto_repair_bluestore_failed_norecov: for i in $(seq 1 10) 2019-12-21T16:01:00.047 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:529: TEST_auto_repair_bluestore_failed_norecov: rados --pool testpool put obj4 td/osd-scrub-repair/ORIGINAL 2019-12-21T16:01:00.081 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:527: TEST_auto_repair_bluestore_failed_norecov: for i in $(seq 1 10) 2019-12-21T16:01:00.081 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:529: TEST_auto_repair_bluestore_failed_norecov: rados --pool testpool put obj5 td/osd-scrub-repair/ORIGINAL 2019-12-21T16:01:00.125 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:527: TEST_auto_repair_bluestore_failed_norecov: for i in $(seq 1 10) 2019-12-21T16:01:00.125 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:529: TEST_auto_repair_bluestore_failed_norecov: rados --pool testpool put obj6 td/osd-scrub-repair/ORIGINAL 2019-12-21T16:01:00.164 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:527: TEST_auto_repair_bluestore_failed_norecov: for i in $(seq 1 10) 2019-12-21T16:01:00.165 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:529: TEST_auto_repair_bluestore_failed_norecov: rados --pool testpool put obj7 td/osd-scrub-repair/ORIGINAL 2019-12-21T16:01:00.200 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:527: TEST_auto_repair_bluestore_failed_norecov: for i in $(seq 1 10) 2019-12-21T16:01:00.200 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:529: TEST_auto_repair_bluestore_failed_norecov: rados --pool testpool put obj8 td/osd-scrub-repair/ORIGINAL 2019-12-21T16:01:00.233 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:527: TEST_auto_repair_bluestore_failed_norecov: for i in $(seq 1 10) 2019-12-21T16:01:00.233 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:529: TEST_auto_repair_bluestore_failed_norecov: rados --pool testpool put obj9 td/osd-scrub-repair/ORIGINAL 2019-12-21T16:01:00.273 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:527: TEST_auto_repair_bluestore_failed_norecov: for i in $(seq 1 10) 2019-12-21T16:01:00.273 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:529: TEST_auto_repair_bluestore_failed_norecov: rados --pool testpool put obj10 td/osd-scrub-repair/ORIGINAL 2019-12-21T16:01:00.309 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:535: TEST_auto_repair_bluestore_failed_norecov: get_not_primary testpool SOMETHING 2019-12-21T16:01:00.309 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1165: get_not_primary: local poolname=testpool 2019-12-21T16:01:00.309 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1166: get_not_primary: local objectname=SOMETHING 2019-12-21T16:01:00.309 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: get_not_primary: get_primary testpool SOMETHING 2019-12-21T16:01:00.310 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_primary: local poolname=testpool 2019-12-21T16:01:00.310 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1133: get_primary: local objectname=SOMETHING 2019-12-21T16:01:00.310 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1135: get_primary: ceph --format json osd map testpool SOMETHING 2019-12-21T16:01:00.310 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1136: get_primary: jq .acting_primary 2019-12-21T16:01:00.538 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: get_not_primary: local primary=1 2019-12-21T16:01:00.538 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1169: get_not_primary: ceph --format json osd map testpool SOMETHING 2019-12-21T16:01:00.539 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1170: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2019-12-21T16:01:00.770 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:535: TEST_auto_repair_bluestore_failed_norecov: objectstore_tool td/osd-scrub-repair 0 obj1 remove 2019-12-21T16:01:00.771 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: objectstore_tool: local dir=td/osd-scrub-repair 2019-12-21T16:01:00.771 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: objectstore_tool: shift 2019-12-21T16:01:00.771 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1235: objectstore_tool: local id=0 2019-12-21T16:01:00.771 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1236: objectstore_tool: shift 2019-12-21T16:01:00.771 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1238: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 obj1 remove 2019-12-21T16:01:00.771 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1205: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2019-12-21T16:01:00.772 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nowait: shift 2019-12-21T16:01:00.772 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1207: _objectstore_tool_nowait: local id=0 2019-12-21T16:01:00.772 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1208: _objectstore_tool_nowait: shift 2019-12-21T16:01:00.772 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1210: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2019-12-21T16:01:00.772 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2019-12-21T16:01:00.772 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2019-12-21T16:01:00.773 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2019-12-21T16:01:00.773 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2019-12-21T16:01:00.773 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2019-12-21T16:01:00.875 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2019-12-21T16:01:00.876 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1212: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 obj1 remove 2019-12-21T16:01:00.876 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1193: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2019-12-21T16:01:00.876 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1194: _objectstore_tool_nodown: shift 2019-12-21T16:01:00.876 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1195: _objectstore_tool_nodown: local id=0 2019-12-21T16:01:00.876 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: _objectstore_tool_nodown: shift 2019-12-21T16:01:00.876 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2019-12-21T16:01:00.876 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj1 remove 2019-12-21T16:01:01.473 INFO:tasks.workunit.client.0.smithi172.stdout:remove #1:ff7b1f36:::obj1:head# 2019-12-21T16:01:01.997 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1213: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2019-12-21T16:01:01.997 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:842: activate_osd: local dir=td/osd-scrub-repair 2019-12-21T16:01:01.997 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: shift 2019-12-21T16:01:01.997 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: local id=0 2019-12-21T16:01:01.998 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: shift 2019-12-21T16:01:01.998 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local osd_data=td/osd-scrub-repair/0 2019-12-21T16:01:01.998 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local 'ceph_args=--fsid=973e3fc1-e63e-4bde-9db6-84c68fd5e93e --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2019-12-21T16:01:01.998 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2019-12-21T16:01:01.998 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-journal-size=100' 2019-12-21T16:01:01.998 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2019-12-21T16:01:01.998 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2019-12-21T16:01:01.999 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2019-12-21T16:01:01.999 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --chdir=' 2019-12-21T16:01:01.999 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+= 2019-12-21T16:01:02.000 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2019-12-21T16:01:02.000 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: get_asok_path 2019-12-21T16:01:02.000 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2019-12-21T16:01:02.000 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2019-12-21T16:01:02.001 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2019-12-21T16:01:02.001 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T16:01:02.001 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.119575 2019-12-21T16:01:02.001 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.119575/$cluster-$name.asok' 2019-12-21T16:01:02.001 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.119575/$cluster-$name.asok' 2019-12-21T16:01:02.001 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --debug-osd=20' 2019-12-21T16:01:02.001 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2019-12-21T16:01:02.002 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2019-12-21T16:01:02.002 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2019-12-21T16:01:02.002 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2019-12-21T16:01:02.002 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2019-12-21T16:01:02.002 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' ' 2019-12-21T16:01:02.002 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2019-12-21T16:01:02.003 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: mkdir -p td/osd-scrub-repair/0 2019-12-21T16:01:02.003 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: echo start osd.0 2019-12-21T16:01:02.003 INFO:tasks.workunit.client.0.smithi172.stderr:start osd.0 2019-12-21T16:01:02.003 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph-osd -i 0 --fsid=973e3fc1-e63e-4bde-9db6-84c68fd5e93e --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.119575/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2019-12-21T16:01:02.003 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: cat td/osd-scrub-repair/0/whoami 2019-12-21T16:01:02.004 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: '[' 0 = 0 ']' 2019-12-21T16:01:02.004 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph osd dump --format=json 2019-12-21T16:01:02.004 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: jq '.flags_set[]' 2019-12-21T16:01:02.004 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: grep -q '"noup"' 2019-12-21T16:01:02.016 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:01:02.015+0000 7f3daf495ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T16:01:02.018 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:01:02.017+0000 7f3daf495ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T16:01:02.020 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:01:02.019+0000 7f3daf495ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T16:01:02.232 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: wait_for_osd up 0 2019-12-21T16:01:02.232 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:914: wait_for_osd: local state=up 2019-12-21T16:01:02.233 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local id=0 2019-12-21T16:01:02.233 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: status=1 2019-12-21T16:01:02.233 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i=0 )) 2019-12-21T16:01:02.233 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i < 300 )) 2019-12-21T16:01:02.233 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: echo 0 2019-12-21T16:01:02.233 INFO:tasks.workunit.client.0.smithi172.stderr:0 2019-12-21T16:01:02.233 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: ceph osd dump 2019-12-21T16:01:02.234 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: grep 'osd.0 up' 2019-12-21T16:01:02.460 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: sleep 1 2019-12-21T16:01:02.536 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:01:02.534+0000 7f3daf495ec0 -1 Falling back to public interface 2019-12-21T16:01:03.318 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:01:03.316+0000 7f3daf495ec0 -1 osd.0 21 log_to_monitors {default=true} 2019-12-21T16:01:03.461 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i++ )) 2019-12-21T16:01:03.461 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i < 300 )) 2019-12-21T16:01:03.461 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: echo 1 2019-12-21T16:01:03.462 INFO:tasks.workunit.client.0.smithi172.stderr:1 2019-12-21T16:01:03.462 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: grep 'osd.0 up' 2019-12-21T16:01:03.462 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: ceph osd dump 2019-12-21T16:01:03.698 INFO:tasks.workunit.client.0.smithi172.stderr:osd.0 up in weight 1 up_from 24 up_thru 0 down_at 22 last_clean_interval [5,21) [v2:127.0.0.1:6802/136951,v1:127.0.0.1:6803/136951] [v2:127.0.0.1:6804/136951,v1:127.0.0.1:6805/136951] exists,up cfa50606-889a-4721-96df-45a2bf205408 2019-12-21T16:01:03.699 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=0 2019-12-21T16:01:03.699 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: break 2019-12-21T16:01:03.699 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: return 0 2019-12-21T16:01:03.699 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1239: objectstore_tool: wait_for_clean 2019-12-21T16:01:03.700 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1548: wait_for_clean: local cmd= 2019-12-21T16:01:03.700 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1549: wait_for_clean: local num_active_clean=-1 2019-12-21T16:01:03.700 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1550: wait_for_clean: local cur_active_clean 2019-12-21T16:01:03.700 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1551: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2019-12-21T16:01:03.701 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1551: wait_for_clean: get_timeout_delays 90 .1 2019-12-21T16:01:03.701 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1494: get_timeout_delays: shopt -q -o xtrace 2019-12-21T16:01:03.701 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1494: get_timeout_delays: echo true 2019-12-21T16:01:03.701 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1494: get_timeout_delays: local trace=true 2019-12-21T16:01:03.701 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1495: get_timeout_delays: true 2019-12-21T16:01:03.701 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1495: get_timeout_delays: shopt -u -o xtrace 2019-12-21T16:01:03.790 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1551: wait_for_clean: local -a delays 2019-12-21T16:01:03.790 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1552: wait_for_clean: local -i loop=0 2019-12-21T16:01:03.790 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1554: wait_for_clean: flush_pg_stats 2019-12-21T16:01:03.790 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1977: flush_pg_stats: local timeout=300 2019-12-21T16:01:03.790 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1979: flush_pg_stats: ceph osd ls 2019-12-21T16:01:04.018 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1979: flush_pg_stats: ids='0 2019-12-21T16:01:04.019 INFO:tasks.workunit.client.0.smithi172.stderr:1 2019-12-21T16:01:04.019 INFO:tasks.workunit.client.0.smithi172.stderr:2' 2019-12-21T16:01:04.019 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1980: flush_pg_stats: seqs= 2019-12-21T16:01:04.019 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: flush_pg_stats: for osd in $ids 2019-12-21T16:01:04.019 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2019-12-21T16:01:04.129 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: seq=103079215106 2019-12-21T16:01:04.129 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: flush_pg_stats: seqs=' 0-103079215106' 2019-12-21T16:01:04.129 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: flush_pg_stats: for osd in $ids 2019-12-21T16:01:04.129 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2019-12-21T16:01:04.239 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: seq=42949672966 2019-12-21T16:01:04.240 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: flush_pg_stats: seqs=' 0-103079215106 1-42949672966' 2019-12-21T16:01:04.240 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: flush_pg_stats: for osd in $ids 2019-12-21T16:01:04.240 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2019-12-21T16:01:04.350 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: seq=64424509445 2019-12-21T16:01:04.351 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: flush_pg_stats: seqs=' 0-103079215106 1-42949672966 2-64424509445' 2019-12-21T16:01:04.351 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: flush_pg_stats: for s in $seqs 2019-12-21T16:01:04.351 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: echo 0-103079215106 2019-12-21T16:01:04.351 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: cut -d - -f 1 2019-12-21T16:01:04.352 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: osd=0 2019-12-21T16:01:04.352 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: echo 0-103079215106 2019-12-21T16:01:04.353 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: cut -d - -f 2 2019-12-21T16:01:04.355 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: seq=103079215106 2019-12-21T16:01:04.355 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1989: flush_pg_stats: echo 'waiting osd.0 seq 103079215106' 2019-12-21T16:01:04.355 INFO:tasks.workunit.client.0.smithi172.stderr:waiting osd.0 seq 103079215106 2019-12-21T16:01:04.355 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 0 2019-12-21T16:01:04.607 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 1 -lt 103079215106 2019-12-21T16:01:04.607 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1991: flush_pg_stats: sleep 1 2019-12-21T16:01:05.608 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1992: flush_pg_stats: '[' 300 -eq 0 ']' 2019-12-21T16:01:05.608 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 0 2019-12-21T16:01:05.845 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 103079215106 -lt 103079215106 2019-12-21T16:01:05.846 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: flush_pg_stats: for s in $seqs 2019-12-21T16:01:05.846 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: echo 1-42949672966 2019-12-21T16:01:05.846 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: cut -d - -f 1 2019-12-21T16:01:05.847 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: osd=1 2019-12-21T16:01:05.847 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: echo 1-42949672966 2019-12-21T16:01:05.847 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: cut -d - -f 2 2019-12-21T16:01:05.848 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: seq=42949672966 2019-12-21T16:01:05.848 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1989: flush_pg_stats: echo 'waiting osd.1 seq 42949672966' 2019-12-21T16:01:05.849 INFO:tasks.workunit.client.0.smithi172.stderr:waiting osd.1 seq 42949672966 2019-12-21T16:01:05.849 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 1 2019-12-21T16:01:06.077 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 42949672966 -lt 42949672966 2019-12-21T16:01:06.077 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: flush_pg_stats: for s in $seqs 2019-12-21T16:01:06.077 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: echo 2-64424509445 2019-12-21T16:01:06.078 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: cut -d - -f 1 2019-12-21T16:01:06.078 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: osd=2 2019-12-21T16:01:06.079 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: echo 2-64424509445 2019-12-21T16:01:06.079 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: cut -d - -f 2 2019-12-21T16:01:06.079 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: seq=64424509445 2019-12-21T16:01:06.080 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1989: flush_pg_stats: echo 'waiting osd.2 seq 64424509445' 2019-12-21T16:01:06.080 INFO:tasks.workunit.client.0.smithi172.stderr:waiting osd.2 seq 64424509445 2019-12-21T16:01:06.080 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 2 2019-12-21T16:01:06.307 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 64424509445 -lt 64424509445 2019-12-21T16:01:06.307 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1555: wait_for_clean: get_num_pgs 2019-12-21T16:01:06.308 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: ceph --format json status 2019-12-21T16:01:06.308 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: jq .pgmap.num_pgs 2019-12-21T16:01:06.576 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1555: wait_for_clean: test 1 == 0 2019-12-21T16:01:06.577 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1559: wait_for_clean: true 2019-12-21T16:01:06.577 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1563: wait_for_clean: get_num_active_clean 2019-12-21T16:01:06.577 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: get_num_active_clean: local expression 2019-12-21T16:01:06.577 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2019-12-21T16:01:06.577 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("stale") | not)' 2019-12-21T16:01:06.577 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: ceph --format json pg dump pgs 2019-12-21T16:01:06.578 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2019-12-21T16:01:06.799 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1563: wait_for_clean: cur_active_clean=1 2019-12-21T16:01:06.800 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1564: wait_for_clean: get_num_pgs 2019-12-21T16:01:06.800 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: ceph --format json status 2019-12-21T16:01:06.801 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: jq .pgmap.num_pgs 2019-12-21T16:01:07.066 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1564: wait_for_clean: test 1 = 1 2019-12-21T16:01:07.067 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1564: wait_for_clean: break 2019-12-21T16:01:07.067 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: return 0 2019-12-21T16:01:07.067 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:536: TEST_auto_repair_bluestore_failed_norecov: get_primary testpool SOMETHING 2019-12-21T16:01:07.067 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_primary: local poolname=testpool 2019-12-21T16:01:07.067 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1133: get_primary: local objectname=SOMETHING 2019-12-21T16:01:07.067 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1135: get_primary: ceph --format json osd map testpool SOMETHING 2019-12-21T16:01:07.068 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1136: get_primary: jq .acting_primary 2019-12-21T16:01:07.314 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:536: TEST_auto_repair_bluestore_failed_norecov: objectstore_tool td/osd-scrub-repair 1 obj1 rm-attr _ 2019-12-21T16:01:07.315 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: objectstore_tool: local dir=td/osd-scrub-repair 2019-12-21T16:01:07.315 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: objectstore_tool: shift 2019-12-21T16:01:07.315 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1235: objectstore_tool: local id=1 2019-12-21T16:01:07.315 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1236: objectstore_tool: shift 2019-12-21T16:01:07.316 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1238: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 obj1 rm-attr _ 2019-12-21T16:01:07.316 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1205: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2019-12-21T16:01:07.316 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nowait: shift 2019-12-21T16:01:07.316 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1207: _objectstore_tool_nowait: local id=1 2019-12-21T16:01:07.317 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1208: _objectstore_tool_nowait: shift 2019-12-21T16:01:07.317 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1210: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2019-12-21T16:01:07.317 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2019-12-21T16:01:07.317 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2019-12-21T16:01:07.317 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2019-12-21T16:01:07.317 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2019-12-21T16:01:07.317 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2019-12-21T16:01:07.418 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2019-12-21T16:01:07.418 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1212: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 obj1 rm-attr _ 2019-12-21T16:01:07.419 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1193: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2019-12-21T16:01:07.419 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1194: _objectstore_tool_nodown: shift 2019-12-21T16:01:07.419 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1195: _objectstore_tool_nodown: local id=1 2019-12-21T16:01:07.419 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: _objectstore_tool_nodown: shift 2019-12-21T16:01:07.419 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2019-12-21T16:01:07.419 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj1 rm-attr _ 2019-12-21T16:01:08.703 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1213: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2019-12-21T16:01:08.703 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:842: activate_osd: local dir=td/osd-scrub-repair 2019-12-21T16:01:08.703 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: shift 2019-12-21T16:01:08.703 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: local id=1 2019-12-21T16:01:08.703 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: shift 2019-12-21T16:01:08.703 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local osd_data=td/osd-scrub-repair/1 2019-12-21T16:01:08.704 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local 'ceph_args=--fsid=973e3fc1-e63e-4bde-9db6-84c68fd5e93e --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2019-12-21T16:01:08.704 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2019-12-21T16:01:08.704 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-journal-size=100' 2019-12-21T16:01:08.704 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2019-12-21T16:01:08.704 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2019-12-21T16:01:08.704 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2019-12-21T16:01:08.704 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --chdir=' 2019-12-21T16:01:08.705 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+= 2019-12-21T16:01:08.705 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2019-12-21T16:01:08.705 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: get_asok_path 2019-12-21T16:01:08.705 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2019-12-21T16:01:08.705 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2019-12-21T16:01:08.705 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2019-12-21T16:01:08.705 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T16:01:08.706 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.119575 2019-12-21T16:01:08.706 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.119575/$cluster-$name.asok' 2019-12-21T16:01:08.706 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.119575/$cluster-$name.asok' 2019-12-21T16:01:08.706 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --debug-osd=20' 2019-12-21T16:01:08.706 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2019-12-21T16:01:08.706 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2019-12-21T16:01:08.706 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2019-12-21T16:01:08.707 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2019-12-21T16:01:08.707 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2019-12-21T16:01:08.707 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' ' 2019-12-21T16:01:08.707 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2019-12-21T16:01:08.707 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: mkdir -p td/osd-scrub-repair/1 2019-12-21T16:01:08.707 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: echo start osd.1 2019-12-21T16:01:08.707 INFO:tasks.workunit.client.0.smithi172.stderr:start osd.1 2019-12-21T16:01:08.708 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph-osd -i 1 --fsid=973e3fc1-e63e-4bde-9db6-84c68fd5e93e --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.119575/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2019-12-21T16:01:08.708 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: cat td/osd-scrub-repair/1/whoami 2019-12-21T16:01:08.708 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: '[' 1 = 1 ']' 2019-12-21T16:01:08.708 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph osd dump --format=json 2019-12-21T16:01:08.708 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: jq '.flags_set[]' 2019-12-21T16:01:08.708 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: grep -q '"noup"' 2019-12-21T16:01:08.725 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:01:08.722+0000 7fa255883ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T16:01:08.727 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:01:08.724+0000 7fa255883ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T16:01:08.730 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:01:08.727+0000 7fa255883ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T16:01:08.937 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: wait_for_osd up 1 2019-12-21T16:01:08.938 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:914: wait_for_osd: local state=up 2019-12-21T16:01:08.938 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local id=1 2019-12-21T16:01:08.938 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: status=1 2019-12-21T16:01:08.938 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i=0 )) 2019-12-21T16:01:08.938 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i < 300 )) 2019-12-21T16:01:08.939 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: echo 0 2019-12-21T16:01:08.939 INFO:tasks.workunit.client.0.smithi172.stderr:0 2019-12-21T16:01:08.939 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: ceph osd dump 2019-12-21T16:01:08.939 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: grep 'osd.1 up' 2019-12-21T16:01:09.162 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: sleep 1 2019-12-21T16:01:09.246 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:01:09.243+0000 7fa255883ec0 -1 Falling back to public interface 2019-12-21T16:01:10.085 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:01:10.083+0000 7fa255883ec0 -1 osd.1 25 log_to_monitors {default=true} 2019-12-21T16:01:10.164 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i++ )) 2019-12-21T16:01:10.164 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i < 300 )) 2019-12-21T16:01:10.164 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: echo 1 2019-12-21T16:01:10.164 INFO:tasks.workunit.client.0.smithi172.stderr:1 2019-12-21T16:01:10.165 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: ceph osd dump 2019-12-21T16:01:10.165 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: grep 'osd.1 up' 2019-12-21T16:01:10.405 INFO:tasks.workunit.client.0.smithi172.stderr:osd.1 up in weight 1 up_from 28 up_thru 28 down_at 26 last_clean_interval [10,25) [v2:127.0.0.1:6810/137639,v1:127.0.0.1:6811/137639] [v2:127.0.0.1:6812/137639,v1:127.0.0.1:6813/137639] exists,up 3a5b611e-245f-49af-9fac-3bc6b41e6466 2019-12-21T16:01:10.405 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=0 2019-12-21T16:01:10.405 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: break 2019-12-21T16:01:10.405 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: return 0 2019-12-21T16:01:10.406 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1239: objectstore_tool: wait_for_clean 2019-12-21T16:01:10.406 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1548: wait_for_clean: local cmd= 2019-12-21T16:01:10.406 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1549: wait_for_clean: local num_active_clean=-1 2019-12-21T16:01:10.406 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1550: wait_for_clean: local cur_active_clean 2019-12-21T16:01:10.406 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1551: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2019-12-21T16:01:10.406 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1551: wait_for_clean: get_timeout_delays 90 .1 2019-12-21T16:01:10.406 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1494: get_timeout_delays: shopt -q -o xtrace 2019-12-21T16:01:10.407 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1494: get_timeout_delays: echo true 2019-12-21T16:01:10.407 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1494: get_timeout_delays: local trace=true 2019-12-21T16:01:10.407 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1495: get_timeout_delays: true 2019-12-21T16:01:10.407 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1495: get_timeout_delays: shopt -u -o xtrace 2019-12-21T16:01:10.497 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1551: wait_for_clean: local -a delays 2019-12-21T16:01:10.497 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1552: wait_for_clean: local -i loop=0 2019-12-21T16:01:10.497 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1554: wait_for_clean: flush_pg_stats 2019-12-21T16:01:10.497 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1977: flush_pg_stats: local timeout=300 2019-12-21T16:01:10.498 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1979: flush_pg_stats: ceph osd ls 2019-12-21T16:01:10.725 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1979: flush_pg_stats: ids='0 2019-12-21T16:01:10.725 INFO:tasks.workunit.client.0.smithi172.stderr:1 2019-12-21T16:01:10.726 INFO:tasks.workunit.client.0.smithi172.stderr:2' 2019-12-21T16:01:10.726 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1980: flush_pg_stats: seqs= 2019-12-21T16:01:10.726 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: flush_pg_stats: for osd in $ids 2019-12-21T16:01:10.726 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2019-12-21T16:01:10.835 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: seq=103079215108 2019-12-21T16:01:10.836 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: flush_pg_stats: seqs=' 0-103079215108' 2019-12-21T16:01:10.836 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: flush_pg_stats: for osd in $ids 2019-12-21T16:01:10.836 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2019-12-21T16:01:10.947 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: seq=120259084290 2019-12-21T16:01:10.947 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: flush_pg_stats: seqs=' 0-103079215108 1-120259084290' 2019-12-21T16:01:10.948 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: flush_pg_stats: for osd in $ids 2019-12-21T16:01:10.948 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2019-12-21T16:01:11.057 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: seq=64424509447 2019-12-21T16:01:11.058 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: flush_pg_stats: seqs=' 0-103079215108 1-120259084290 2-64424509447' 2019-12-21T16:01:11.058 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: flush_pg_stats: for s in $seqs 2019-12-21T16:01:11.058 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: echo 0-103079215108 2019-12-21T16:01:11.058 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: cut -d - -f 1 2019-12-21T16:01:11.059 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: osd=0 2019-12-21T16:01:11.059 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: echo 0-103079215108 2019-12-21T16:01:11.059 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: cut -d - -f 2 2019-12-21T16:01:11.060 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: seq=103079215108 2019-12-21T16:01:11.060 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1989: flush_pg_stats: echo 'waiting osd.0 seq 103079215108' 2019-12-21T16:01:11.060 INFO:tasks.workunit.client.0.smithi172.stderr:waiting osd.0 seq 103079215108 2019-12-21T16:01:11.061 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 0 2019-12-21T16:01:11.286 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 103079215107 -lt 103079215108 2019-12-21T16:01:11.286 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1991: flush_pg_stats: sleep 1 2019-12-21T16:01:12.287 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1992: flush_pg_stats: '[' 300 -eq 0 ']' 2019-12-21T16:01:12.287 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 0 2019-12-21T16:01:12.524 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 103079215108 -lt 103079215108 2019-12-21T16:01:12.524 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: flush_pg_stats: for s in $seqs 2019-12-21T16:01:12.524 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: echo 1-120259084290 2019-12-21T16:01:12.525 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: cut -d - -f 1 2019-12-21T16:01:12.525 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: osd=1 2019-12-21T16:01:12.525 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: echo 1-120259084290 2019-12-21T16:01:12.526 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: cut -d - -f 2 2019-12-21T16:01:12.526 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: seq=120259084290 2019-12-21T16:01:12.526 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1989: flush_pg_stats: echo 'waiting osd.1 seq 120259084290' 2019-12-21T16:01:12.527 INFO:tasks.workunit.client.0.smithi172.stderr:waiting osd.1 seq 120259084290 2019-12-21T16:01:12.527 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 1 2019-12-21T16:01:12.754 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 120259084290 -lt 120259084290 2019-12-21T16:01:12.755 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: flush_pg_stats: for s in $seqs 2019-12-21T16:01:12.755 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: echo 2-64424509447 2019-12-21T16:01:12.755 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: cut -d - -f 1 2019-12-21T16:01:12.756 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: osd=2 2019-12-21T16:01:12.756 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: echo 2-64424509447 2019-12-21T16:01:12.756 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: cut -d - -f 2 2019-12-21T16:01:12.757 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: seq=64424509447 2019-12-21T16:01:12.757 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1989: flush_pg_stats: echo 'waiting osd.2 seq 64424509447' 2019-12-21T16:01:12.757 INFO:tasks.workunit.client.0.smithi172.stderr:waiting osd.2 seq 64424509447 2019-12-21T16:01:12.758 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 2 2019-12-21T16:01:12.985 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 64424509447 -lt 64424509447 2019-12-21T16:01:12.985 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1555: wait_for_clean: get_num_pgs 2019-12-21T16:01:12.985 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: ceph --format json status 2019-12-21T16:01:12.986 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: jq .pgmap.num_pgs 2019-12-21T16:01:13.253 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1555: wait_for_clean: test 1 == 0 2019-12-21T16:01:13.253 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1559: wait_for_clean: true 2019-12-21T16:01:13.253 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1563: wait_for_clean: get_num_active_clean 2019-12-21T16:01:13.253 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: get_num_active_clean: local expression 2019-12-21T16:01:13.253 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2019-12-21T16:01:13.254 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("stale") | not)' 2019-12-21T16:01:13.254 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: ceph --format json pg dump pgs 2019-12-21T16:01:13.254 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2019-12-21T16:01:13.477 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1563: wait_for_clean: cur_active_clean=1 2019-12-21T16:01:13.477 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1564: wait_for_clean: get_num_pgs 2019-12-21T16:01:13.477 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: ceph --format json status 2019-12-21T16:01:13.478 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: jq .pgmap.num_pgs 2019-12-21T16:01:13.748 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1564: wait_for_clean: test 1 = 1 2019-12-21T16:01:13.748 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1564: wait_for_clean: break 2019-12-21T16:01:13.749 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: return 0 2019-12-21T16:01:13.749 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:538: TEST_auto_repair_bluestore_failed_norecov: get_not_primary testpool SOMETHING 2019-12-21T16:01:13.749 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1165: get_not_primary: local poolname=testpool 2019-12-21T16:01:13.749 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1166: get_not_primary: local objectname=SOMETHING 2019-12-21T16:01:13.749 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: get_not_primary: get_primary testpool SOMETHING 2019-12-21T16:01:13.749 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_primary: local poolname=testpool 2019-12-21T16:01:13.749 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1133: get_primary: local objectname=SOMETHING 2019-12-21T16:01:13.750 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1135: get_primary: ceph --format json osd map testpool SOMETHING 2019-12-21T16:01:13.750 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1136: get_primary: jq .acting_primary 2019-12-21T16:01:13.981 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: get_not_primary: local primary=1 2019-12-21T16:01:13.981 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1169: get_not_primary: ceph --format json osd map testpool SOMETHING 2019-12-21T16:01:13.981 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1170: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2019-12-21T16:01:14.212 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:538: TEST_auto_repair_bluestore_failed_norecov: objectstore_tool td/osd-scrub-repair 0 obj2 remove 2019-12-21T16:01:14.212 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: objectstore_tool: local dir=td/osd-scrub-repair 2019-12-21T16:01:14.212 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: objectstore_tool: shift 2019-12-21T16:01:14.212 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1235: objectstore_tool: local id=0 2019-12-21T16:01:14.213 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1236: objectstore_tool: shift 2019-12-21T16:01:14.213 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1238: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 obj2 remove 2019-12-21T16:01:14.213 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1205: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2019-12-21T16:01:14.213 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nowait: shift 2019-12-21T16:01:14.213 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1207: _objectstore_tool_nowait: local id=0 2019-12-21T16:01:14.213 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1208: _objectstore_tool_nowait: shift 2019-12-21T16:01:14.213 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1210: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2019-12-21T16:01:14.214 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2019-12-21T16:01:14.214 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2019-12-21T16:01:14.214 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2019-12-21T16:01:14.214 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2019-12-21T16:01:14.214 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2019-12-21T16:01:14.317 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2019-12-21T16:01:14.317 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1212: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 obj2 remove 2019-12-21T16:01:14.317 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1193: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2019-12-21T16:01:14.317 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1194: _objectstore_tool_nodown: shift 2019-12-21T16:01:14.317 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1195: _objectstore_tool_nodown: local id=0 2019-12-21T16:01:14.318 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: _objectstore_tool_nodown: shift 2019-12-21T16:01:14.318 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2019-12-21T16:01:14.318 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj2 remove 2019-12-21T16:01:14.963 INFO:tasks.workunit.client.0.smithi172.stdout:remove #1:104778fc:::obj2:head# 2019-12-21T16:01:15.481 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1213: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2019-12-21T16:01:15.481 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:842: activate_osd: local dir=td/osd-scrub-repair 2019-12-21T16:01:15.481 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: shift 2019-12-21T16:01:15.481 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: local id=0 2019-12-21T16:01:15.482 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: shift 2019-12-21T16:01:15.482 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local osd_data=td/osd-scrub-repair/0 2019-12-21T16:01:15.482 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local 'ceph_args=--fsid=973e3fc1-e63e-4bde-9db6-84c68fd5e93e --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2019-12-21T16:01:15.482 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2019-12-21T16:01:15.482 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-journal-size=100' 2019-12-21T16:01:15.482 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2019-12-21T16:01:15.483 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2019-12-21T16:01:15.483 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2019-12-21T16:01:15.483 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --chdir=' 2019-12-21T16:01:15.483 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+= 2019-12-21T16:01:15.483 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2019-12-21T16:01:15.483 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: get_asok_path 2019-12-21T16:01:15.483 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2019-12-21T16:01:15.484 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2019-12-21T16:01:15.484 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2019-12-21T16:01:15.484 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T16:01:15.484 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.119575 2019-12-21T16:01:15.484 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.119575/$cluster-$name.asok' 2019-12-21T16:01:15.484 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.119575/$cluster-$name.asok' 2019-12-21T16:01:15.485 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --debug-osd=20' 2019-12-21T16:01:15.485 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2019-12-21T16:01:15.485 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2019-12-21T16:01:15.485 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2019-12-21T16:01:15.485 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2019-12-21T16:01:15.485 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2019-12-21T16:01:15.485 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' ' 2019-12-21T16:01:15.486 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2019-12-21T16:01:15.486 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: mkdir -p td/osd-scrub-repair/0 2019-12-21T16:01:15.487 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: echo start osd.0 2019-12-21T16:01:15.487 INFO:tasks.workunit.client.0.smithi172.stderr:start osd.0 2019-12-21T16:01:15.487 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph-osd -i 0 --fsid=973e3fc1-e63e-4bde-9db6-84c68fd5e93e --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.119575/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2019-12-21T16:01:15.487 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: cat td/osd-scrub-repair/0/whoami 2019-12-21T16:01:15.487 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: '[' 0 = 0 ']' 2019-12-21T16:01:15.487 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph osd dump --format=json 2019-12-21T16:01:15.488 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: jq '.flags_set[]' 2019-12-21T16:01:15.488 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: grep -q '"noup"' 2019-12-21T16:01:15.506 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:01:15.503+0000 7f8c4e334ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T16:01:15.508 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:01:15.505+0000 7f8c4e334ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T16:01:15.509 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:01:15.506+0000 7f8c4e334ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T16:01:15.722 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: wait_for_osd up 0 2019-12-21T16:01:15.723 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:914: wait_for_osd: local state=up 2019-12-21T16:01:15.723 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local id=0 2019-12-21T16:01:15.723 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: status=1 2019-12-21T16:01:15.723 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i=0 )) 2019-12-21T16:01:15.723 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i < 300 )) 2019-12-21T16:01:15.723 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: echo 0 2019-12-21T16:01:15.723 INFO:tasks.workunit.client.0.smithi172.stderr:0 2019-12-21T16:01:15.724 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: ceph osd dump 2019-12-21T16:01:15.724 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: grep 'osd.0 up' 2019-12-21T16:01:15.965 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: sleep 1 2019-12-21T16:01:16.025 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:01:16.023+0000 7f8c4e334ec0 -1 Falling back to public interface 2019-12-21T16:01:16.728 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:01:16.725+0000 7f8c4e334ec0 -1 osd.0 29 log_to_monitors {default=true} 2019-12-21T16:01:16.967 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i++ )) 2019-12-21T16:01:16.967 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i < 300 )) 2019-12-21T16:01:16.967 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: echo 1 2019-12-21T16:01:16.967 INFO:tasks.workunit.client.0.smithi172.stderr:1 2019-12-21T16:01:16.967 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: ceph osd dump 2019-12-21T16:01:16.968 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: grep 'osd.0 up' 2019-12-21T16:01:17.214 INFO:tasks.workunit.client.0.smithi172.stderr:osd.0 up in weight 1 up_from 32 up_thru 26 down_at 30 last_clean_interval [24,29) [v2:127.0.0.1:6802/138354,v1:127.0.0.1:6803/138354] [v2:127.0.0.1:6804/138354,v1:127.0.0.1:6805/138354] exists,up cfa50606-889a-4721-96df-45a2bf205408 2019-12-21T16:01:17.214 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=0 2019-12-21T16:01:17.214 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: break 2019-12-21T16:01:17.214 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: return 0 2019-12-21T16:01:17.214 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1239: objectstore_tool: wait_for_clean 2019-12-21T16:01:17.214 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1548: wait_for_clean: local cmd= 2019-12-21T16:01:17.215 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1549: wait_for_clean: local num_active_clean=-1 2019-12-21T16:01:17.215 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1550: wait_for_clean: local cur_active_clean 2019-12-21T16:01:17.215 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1551: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2019-12-21T16:01:17.215 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1551: wait_for_clean: get_timeout_delays 90 .1 2019-12-21T16:01:17.215 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1494: get_timeout_delays: shopt -q -o xtrace 2019-12-21T16:01:17.215 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1494: get_timeout_delays: echo true 2019-12-21T16:01:17.215 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1494: get_timeout_delays: local trace=true 2019-12-21T16:01:17.216 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1495: get_timeout_delays: true 2019-12-21T16:01:17.216 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1495: get_timeout_delays: shopt -u -o xtrace 2019-12-21T16:01:17.306 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1551: wait_for_clean: local -a delays 2019-12-21T16:01:17.307 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1552: wait_for_clean: local -i loop=0 2019-12-21T16:01:17.307 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1554: wait_for_clean: flush_pg_stats 2019-12-21T16:01:17.307 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1977: flush_pg_stats: local timeout=300 2019-12-21T16:01:17.307 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1979: flush_pg_stats: ceph osd ls 2019-12-21T16:01:17.533 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1979: flush_pg_stats: ids='0 2019-12-21T16:01:17.533 INFO:tasks.workunit.client.0.smithi172.stderr:1 2019-12-21T16:01:17.533 INFO:tasks.workunit.client.0.smithi172.stderr:2' 2019-12-21T16:01:17.533 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1980: flush_pg_stats: seqs= 2019-12-21T16:01:17.534 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: flush_pg_stats: for osd in $ids 2019-12-21T16:01:17.534 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2019-12-21T16:01:17.643 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: seq=137438953474 2019-12-21T16:01:17.644 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: flush_pg_stats: seqs=' 0-137438953474' 2019-12-21T16:01:17.644 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: flush_pg_stats: for osd in $ids 2019-12-21T16:01:17.644 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2019-12-21T16:01:17.775 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: seq=120259084292 2019-12-21T16:01:17.775 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: flush_pg_stats: seqs=' 0-137438953474 1-120259084292' 2019-12-21T16:01:17.776 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: flush_pg_stats: for osd in $ids 2019-12-21T16:01:17.776 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2019-12-21T16:01:17.887 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: seq=64424509449 2019-12-21T16:01:17.888 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: flush_pg_stats: seqs=' 0-137438953474 1-120259084292 2-64424509449' 2019-12-21T16:01:17.888 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: flush_pg_stats: for s in $seqs 2019-12-21T16:01:17.888 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: echo 0-137438953474 2019-12-21T16:01:17.888 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: cut -d - -f 1 2019-12-21T16:01:17.889 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: osd=0 2019-12-21T16:01:17.889 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: echo 0-137438953474 2019-12-21T16:01:17.890 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: cut -d - -f 2 2019-12-21T16:01:17.890 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: seq=137438953474 2019-12-21T16:01:17.891 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1989: flush_pg_stats: echo 'waiting osd.0 seq 137438953474' 2019-12-21T16:01:17.891 INFO:tasks.workunit.client.0.smithi172.stderr:waiting osd.0 seq 137438953474 2019-12-21T16:01:17.891 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 0 2019-12-21T16:01:18.119 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 137438953474 -lt 137438953474 2019-12-21T16:01:18.119 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: flush_pg_stats: for s in $seqs 2019-12-21T16:01:18.120 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: echo 1-120259084292 2019-12-21T16:01:18.120 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: cut -d - -f 1 2019-12-21T16:01:18.120 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: osd=1 2019-12-21T16:01:18.121 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: echo 1-120259084292 2019-12-21T16:01:18.121 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: cut -d - -f 2 2019-12-21T16:01:18.122 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: seq=120259084292 2019-12-21T16:01:18.122 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1989: flush_pg_stats: echo 'waiting osd.1 seq 120259084292' 2019-12-21T16:01:18.122 INFO:tasks.workunit.client.0.smithi172.stderr:waiting osd.1 seq 120259084292 2019-12-21T16:01:18.123 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 1 2019-12-21T16:01:18.351 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 120259084291 -lt 120259084292 2019-12-21T16:01:18.351 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1991: flush_pg_stats: sleep 1 2019-12-21T16:01:19.352 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1992: flush_pg_stats: '[' 300 -eq 0 ']' 2019-12-21T16:01:19.353 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 1 2019-12-21T16:01:19.589 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 120259084291 -lt 120259084292 2019-12-21T16:01:19.589 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1991: flush_pg_stats: sleep 1 2019-12-21T16:01:20.590 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1992: flush_pg_stats: '[' 299 -eq 0 ']' 2019-12-21T16:01:20.590 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 1 2019-12-21T16:01:20.826 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 120259084292 -lt 120259084292 2019-12-21T16:01:20.826 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: flush_pg_stats: for s in $seqs 2019-12-21T16:01:20.827 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: echo 2-64424509449 2019-12-21T16:01:20.827 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: cut -d - -f 1 2019-12-21T16:01:20.827 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: osd=2 2019-12-21T16:01:20.828 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: echo 2-64424509449 2019-12-21T16:01:20.828 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: cut -d - -f 2 2019-12-21T16:01:20.829 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: seq=64424509449 2019-12-21T16:01:20.829 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1989: flush_pg_stats: echo 'waiting osd.2 seq 64424509449' 2019-12-21T16:01:20.829 INFO:tasks.workunit.client.0.smithi172.stderr:waiting osd.2 seq 64424509449 2019-12-21T16:01:20.829 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 2 2019-12-21T16:01:21.056 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 64424509450 -lt 64424509449 2019-12-21T16:01:21.057 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1555: wait_for_clean: get_num_pgs 2019-12-21T16:01:21.057 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: ceph --format json status 2019-12-21T16:01:21.057 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: jq .pgmap.num_pgs 2019-12-21T16:01:21.324 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1555: wait_for_clean: test 1 == 0 2019-12-21T16:01:21.325 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1559: wait_for_clean: true 2019-12-21T16:01:21.325 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1563: wait_for_clean: get_num_active_clean 2019-12-21T16:01:21.325 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: get_num_active_clean: local expression 2019-12-21T16:01:21.325 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2019-12-21T16:01:21.325 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("stale") | not)' 2019-12-21T16:01:21.325 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: ceph --format json pg dump pgs 2019-12-21T16:01:21.326 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2019-12-21T16:01:21.547 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1563: wait_for_clean: cur_active_clean=1 2019-12-21T16:01:21.547 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1564: wait_for_clean: get_num_pgs 2019-12-21T16:01:21.547 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: ceph --format json status 2019-12-21T16:01:21.548 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: jq .pgmap.num_pgs 2019-12-21T16:01:21.815 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1564: wait_for_clean: test 1 = 1 2019-12-21T16:01:21.815 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1564: wait_for_clean: break 2019-12-21T16:01:21.815 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: return 0 2019-12-21T16:01:21.816 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:539: TEST_auto_repair_bluestore_failed_norecov: get_primary testpool SOMETHING 2019-12-21T16:01:21.816 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_primary: local poolname=testpool 2019-12-21T16:01:21.817 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1133: get_primary: local objectname=SOMETHING 2019-12-21T16:01:21.817 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1135: get_primary: ceph --format json osd map testpool SOMETHING 2019-12-21T16:01:21.817 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1136: get_primary: jq .acting_primary 2019-12-21T16:01:22.047 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:539: TEST_auto_repair_bluestore_failed_norecov: objectstore_tool td/osd-scrub-repair 1 obj2 rm-attr _ 2019-12-21T16:01:22.047 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: objectstore_tool: local dir=td/osd-scrub-repair 2019-12-21T16:01:22.047 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: objectstore_tool: shift 2019-12-21T16:01:22.047 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1235: objectstore_tool: local id=1 2019-12-21T16:01:22.047 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1236: objectstore_tool: shift 2019-12-21T16:01:22.047 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1238: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 obj2 rm-attr _ 2019-12-21T16:01:22.048 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1205: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2019-12-21T16:01:22.048 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nowait: shift 2019-12-21T16:01:22.048 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1207: _objectstore_tool_nowait: local id=1 2019-12-21T16:01:22.048 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1208: _objectstore_tool_nowait: shift 2019-12-21T16:01:22.048 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1210: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2019-12-21T16:01:22.048 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2019-12-21T16:01:22.048 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2019-12-21T16:01:22.049 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2019-12-21T16:01:22.049 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2019-12-21T16:01:22.049 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2019-12-21T16:01:22.151 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2019-12-21T16:01:22.152 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1212: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 obj2 rm-attr _ 2019-12-21T16:01:22.152 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1193: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2019-12-21T16:01:22.152 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1194: _objectstore_tool_nodown: shift 2019-12-21T16:01:22.152 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1195: _objectstore_tool_nodown: local id=1 2019-12-21T16:01:22.152 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: _objectstore_tool_nodown: shift 2019-12-21T16:01:22.152 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2019-12-21T16:01:22.152 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj2 rm-attr _ 2019-12-21T16:01:22.808 INFO:tasks.workunit.client.0.smithi172.stderr:Error getting attr on : 1.0_head,#1:ff7b1f36:::obj1:head#, (61) No data available 2019-12-21T16:01:23.327 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1213: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2019-12-21T16:01:23.327 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:842: activate_osd: local dir=td/osd-scrub-repair 2019-12-21T16:01:23.328 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: shift 2019-12-21T16:01:23.328 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: local id=1 2019-12-21T16:01:23.328 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: shift 2019-12-21T16:01:23.328 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local osd_data=td/osd-scrub-repair/1 2019-12-21T16:01:23.328 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local 'ceph_args=--fsid=973e3fc1-e63e-4bde-9db6-84c68fd5e93e --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2019-12-21T16:01:23.328 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2019-12-21T16:01:23.328 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-journal-size=100' 2019-12-21T16:01:23.329 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2019-12-21T16:01:23.329 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2019-12-21T16:01:23.329 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2019-12-21T16:01:23.329 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --chdir=' 2019-12-21T16:01:23.329 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+= 2019-12-21T16:01:23.329 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2019-12-21T16:01:23.329 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: get_asok_path 2019-12-21T16:01:23.330 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2019-12-21T16:01:23.330 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2019-12-21T16:01:23.330 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2019-12-21T16:01:23.330 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T16:01:23.330 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.119575 2019-12-21T16:01:23.330 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.119575/$cluster-$name.asok' 2019-12-21T16:01:23.330 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.119575/$cluster-$name.asok' 2019-12-21T16:01:23.331 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --debug-osd=20' 2019-12-21T16:01:23.331 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2019-12-21T16:01:23.331 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2019-12-21T16:01:23.331 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2019-12-21T16:01:23.331 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2019-12-21T16:01:23.331 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2019-12-21T16:01:23.331 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' ' 2019-12-21T16:01:23.332 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2019-12-21T16:01:23.332 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: mkdir -p td/osd-scrub-repair/1 2019-12-21T16:01:23.332 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: echo start osd.1 2019-12-21T16:01:23.332 INFO:tasks.workunit.client.0.smithi172.stderr:start osd.1 2019-12-21T16:01:23.333 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph-osd -i 1 --fsid=973e3fc1-e63e-4bde-9db6-84c68fd5e93e --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.119575/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2019-12-21T16:01:23.333 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: cat td/osd-scrub-repair/1/whoami 2019-12-21T16:01:23.333 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: '[' 1 = 1 ']' 2019-12-21T16:01:23.333 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph osd dump --format=json 2019-12-21T16:01:23.334 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: jq '.flags_set[]' 2019-12-21T16:01:23.334 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: grep -q '"noup"' 2019-12-21T16:01:23.350 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:01:23.347+0000 7fa0e08b2ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T16:01:23.352 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:01:23.350+0000 7fa0e08b2ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T16:01:23.355 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:01:23.352+0000 7fa0e08b2ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T16:01:23.563 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: wait_for_osd up 1 2019-12-21T16:01:23.564 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:914: wait_for_osd: local state=up 2019-12-21T16:01:23.564 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local id=1 2019-12-21T16:01:23.564 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: status=1 2019-12-21T16:01:23.564 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i=0 )) 2019-12-21T16:01:23.564 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i < 300 )) 2019-12-21T16:01:23.564 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: echo 0 2019-12-21T16:01:23.565 INFO:tasks.workunit.client.0.smithi172.stderr:0 2019-12-21T16:01:23.565 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: ceph osd dump 2019-12-21T16:01:23.565 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: grep 'osd.1 up' 2019-12-21T16:01:23.793 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: sleep 1 2019-12-21T16:01:23.870 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:01:23.868+0000 7fa0e08b2ec0 -1 Falling back to public interface 2019-12-21T16:01:24.639 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:01:24.636+0000 7fa0e08b2ec0 -1 osd.1 33 log_to_monitors {default=true} 2019-12-21T16:01:24.795 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i++ )) 2019-12-21T16:01:24.795 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i < 300 )) 2019-12-21T16:01:24.795 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: echo 1 2019-12-21T16:01:24.795 INFO:tasks.workunit.client.0.smithi172.stderr:1 2019-12-21T16:01:24.796 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: grep 'osd.1 up' 2019-12-21T16:01:24.796 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: ceph osd dump 2019-12-21T16:01:25.080 INFO:tasks.workunit.client.0.smithi172.stderr:osd.1 up in weight 1 up_from 36 up_thru 36 down_at 34 last_clean_interval [28,33) [v2:127.0.0.1:6810/139069,v1:127.0.0.1:6811/139069] [v2:127.0.0.1:6812/139069,v1:127.0.0.1:6813/139069] exists,up 3a5b611e-245f-49af-9fac-3bc6b41e6466 2019-12-21T16:01:25.080 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=0 2019-12-21T16:01:25.081 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: break 2019-12-21T16:01:25.081 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: return 0 2019-12-21T16:01:25.081 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1239: objectstore_tool: wait_for_clean 2019-12-21T16:01:25.081 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1548: wait_for_clean: local cmd= 2019-12-21T16:01:25.082 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1549: wait_for_clean: local num_active_clean=-1 2019-12-21T16:01:25.082 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1550: wait_for_clean: local cur_active_clean 2019-12-21T16:01:25.082 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1551: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2019-12-21T16:01:25.082 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1551: wait_for_clean: get_timeout_delays 90 .1 2019-12-21T16:01:25.083 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1494: get_timeout_delays: shopt -q -o xtrace 2019-12-21T16:01:25.083 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1494: get_timeout_delays: echo true 2019-12-21T16:01:25.083 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1494: get_timeout_delays: local trace=true 2019-12-21T16:01:25.083 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1495: get_timeout_delays: true 2019-12-21T16:01:25.084 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1495: get_timeout_delays: shopt -u -o xtrace 2019-12-21T16:01:25.172 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1551: wait_for_clean: local -a delays 2019-12-21T16:01:25.172 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1552: wait_for_clean: local -i loop=0 2019-12-21T16:01:25.172 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1554: wait_for_clean: flush_pg_stats 2019-12-21T16:01:25.173 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1977: flush_pg_stats: local timeout=300 2019-12-21T16:01:25.173 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1979: flush_pg_stats: ceph osd ls 2019-12-21T16:01:25.398 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1979: flush_pg_stats: ids='0 2019-12-21T16:01:25.398 INFO:tasks.workunit.client.0.smithi172.stderr:1 2019-12-21T16:01:25.399 INFO:tasks.workunit.client.0.smithi172.stderr:2' 2019-12-21T16:01:25.399 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1980: flush_pg_stats: seqs= 2019-12-21T16:01:25.399 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: flush_pg_stats: for osd in $ids 2019-12-21T16:01:25.399 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2019-12-21T16:01:25.516 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: seq=137438953476 2019-12-21T16:01:25.516 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: flush_pg_stats: seqs=' 0-137438953476' 2019-12-21T16:01:25.516 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: flush_pg_stats: for osd in $ids 2019-12-21T16:01:25.517 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2019-12-21T16:01:25.627 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: seq=154618822658 2019-12-21T16:01:25.627 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: flush_pg_stats: seqs=' 0-137438953476 1-154618822658' 2019-12-21T16:01:25.627 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: flush_pg_stats: for osd in $ids 2019-12-21T16:01:25.628 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2019-12-21T16:01:25.738 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: seq=64424509452 2019-12-21T16:01:25.738 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: flush_pg_stats: seqs=' 0-137438953476 1-154618822658 2-64424509452' 2019-12-21T16:01:25.738 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: flush_pg_stats: for s in $seqs 2019-12-21T16:01:25.738 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: echo 0-137438953476 2019-12-21T16:01:25.739 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: cut -d - -f 1 2019-12-21T16:01:25.740 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: osd=0 2019-12-21T16:01:25.740 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: echo 0-137438953476 2019-12-21T16:01:25.741 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: cut -d - -f 2 2019-12-21T16:01:25.741 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: seq=137438953476 2019-12-21T16:01:25.741 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1989: flush_pg_stats: echo 'waiting osd.0 seq 137438953476' 2019-12-21T16:01:25.741 INFO:tasks.workunit.client.0.smithi172.stderr:waiting osd.0 seq 137438953476 2019-12-21T16:01:25.742 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 0 2019-12-21T16:01:25.968 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 137438953476 -lt 137438953476 2019-12-21T16:01:25.968 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: flush_pg_stats: for s in $seqs 2019-12-21T16:01:25.969 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: echo 1-154618822658 2019-12-21T16:01:25.969 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: cut -d - -f 1 2019-12-21T16:01:25.969 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: osd=1 2019-12-21T16:01:25.970 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: echo 1-154618822658 2019-12-21T16:01:25.970 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: cut -d - -f 2 2019-12-21T16:01:25.971 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: seq=154618822658 2019-12-21T16:01:25.971 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1989: flush_pg_stats: echo 'waiting osd.1 seq 154618822658' 2019-12-21T16:01:25.971 INFO:tasks.workunit.client.0.smithi172.stderr:waiting osd.1 seq 154618822658 2019-12-21T16:01:25.971 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 1 2019-12-21T16:01:26.201 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 154618822658 -lt 154618822658 2019-12-21T16:01:26.202 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: flush_pg_stats: for s in $seqs 2019-12-21T16:01:26.202 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: echo 2-64424509452 2019-12-21T16:01:26.203 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: cut -d - -f 1 2019-12-21T16:01:26.203 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: osd=2 2019-12-21T16:01:26.203 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: echo 2-64424509452 2019-12-21T16:01:26.204 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: cut -d - -f 2 2019-12-21T16:01:26.204 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: seq=64424509452 2019-12-21T16:01:26.205 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1989: flush_pg_stats: echo 'waiting osd.2 seq 64424509452' 2019-12-21T16:01:26.205 INFO:tasks.workunit.client.0.smithi172.stderr:waiting osd.2 seq 64424509452 2019-12-21T16:01:26.205 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 2 2019-12-21T16:01:26.433 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 64424509451 -lt 64424509452 2019-12-21T16:01:26.433 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1991: flush_pg_stats: sleep 1 2019-12-21T16:01:27.434 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1992: flush_pg_stats: '[' 300 -eq 0 ']' 2019-12-21T16:01:27.434 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 2 2019-12-21T16:01:27.670 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 64424509451 -lt 64424509452 2019-12-21T16:01:27.670 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1991: flush_pg_stats: sleep 1 2019-12-21T16:01:28.671 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1992: flush_pg_stats: '[' 299 -eq 0 ']' 2019-12-21T16:01:28.672 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 2 2019-12-21T16:01:28.907 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 64424509452 -lt 64424509452 2019-12-21T16:01:28.908 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1555: wait_for_clean: get_num_pgs 2019-12-21T16:01:28.908 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: ceph --format json status 2019-12-21T16:01:28.909 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: jq .pgmap.num_pgs 2019-12-21T16:01:29.193 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1555: wait_for_clean: test 1 == 0 2019-12-21T16:01:29.194 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1559: wait_for_clean: true 2019-12-21T16:01:29.194 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1563: wait_for_clean: get_num_active_clean 2019-12-21T16:01:29.194 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: get_num_active_clean: local expression 2019-12-21T16:01:29.194 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2019-12-21T16:01:29.194 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("stale") | not)' 2019-12-21T16:01:29.194 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: ceph --format json pg dump pgs 2019-12-21T16:01:29.195 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2019-12-21T16:01:29.415 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1563: wait_for_clean: cur_active_clean=1 2019-12-21T16:01:29.416 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1564: wait_for_clean: get_num_pgs 2019-12-21T16:01:29.416 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: ceph --format json status 2019-12-21T16:01:29.416 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: jq .pgmap.num_pgs 2019-12-21T16:01:29.684 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1564: wait_for_clean: test 1 = 1 2019-12-21T16:01:29.684 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1564: wait_for_clean: break 2019-12-21T16:01:29.684 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: return 0 2019-12-21T16:01:29.684 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:541: TEST_auto_repair_bluestore_failed_norecov: get_pg testpool obj1 2019-12-21T16:01:29.684 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1029: get_pg: local poolname=testpool 2019-12-21T16:01:29.685 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1030: get_pg: local objectname=obj1 2019-12-21T16:01:29.685 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1032: get_pg: ceph --format json osd map testpool obj1 2019-12-21T16:01:29.685 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1032: get_pg: jq -r .pgid 2019-12-21T16:01:29.915 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:541: TEST_auto_repair_bluestore_failed_norecov: local pgid=1.0 2019-12-21T16:01:29.916 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:542: TEST_auto_repair_bluestore_failed_norecov: get_primary testpool obj1 2019-12-21T16:01:29.916 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_primary: local poolname=testpool 2019-12-21T16:01:29.916 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1133: get_primary: local objectname=obj1 2019-12-21T16:01:29.916 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1135: get_primary: ceph --format json osd map testpool obj1 2019-12-21T16:01:29.916 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1136: get_primary: jq .acting_primary 2019-12-21T16:01:30.145 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:542: TEST_auto_repair_bluestore_failed_norecov: local primary=1 2019-12-21T16:01:30.146 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:543: TEST_auto_repair_bluestore_failed_norecov: get_last_scrub_stamp 1.0 2019-12-21T16:01:30.146 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.0 2019-12-21T16:01:30.146 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T16:01:30.146 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T16:01:30.146 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2019-12-21T16:01:30.368 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:543: TEST_auto_repair_bluestore_failed_norecov: local last_scrub_stamp=2019-12-21T16:00:54.978785+0000 2019-12-21T16:01:30.369 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:544: TEST_auto_repair_bluestore_failed_norecov: ceph tell 1.0 deep_scrub 2019-12-21T16:01:30.469 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T16:01:30.470 INFO:tasks.workunit.client.0.smithi172.stdout: "deep": true, 2019-12-21T16:01:30.470 INFO:tasks.workunit.client.0.smithi172.stdout: "stamp": "2019-12-14T15:59:50.468727+0000" 2019-12-21T16:01:30.470 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T16:01:30.478 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:545: TEST_auto_repair_bluestore_failed_norecov: ceph tell 1.0 scrub 2019-12-21T16:01:30.584 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T16:01:30.584 INFO:tasks.workunit.client.0.smithi172.stdout: "deep": false, 2019-12-21T16:01:30.584 INFO:tasks.workunit.client.0.smithi172.stdout: "stamp": "2019-12-14T15:59:50.582902+0000" 2019-12-21T16:01:30.584 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T16:01:30.593 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:548: TEST_auto_repair_bluestore_failed_norecov: wait_for_scrub 1.0 2019-12-21T16:00:54.978785+0000 2019-12-21T16:01:30.593 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1789: wait_for_scrub: local pgid=1.0 2019-12-21T16:01:30.593 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1790: wait_for_scrub: local last_scrub=2019-12-21T16:00:54.978785+0000 2019-12-21T16:01:30.594 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1791: wait_for_scrub: local sname=last_scrub_stamp 2019-12-21T16:01:30.594 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i=0 )) 2019-12-21T16:01:30.594 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T16:01:30.594 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2019-12-21T16:01:30.594 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.0 2019-12-21T16:01:30.594 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T16:01:30.594 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T16:01:30.595 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2019-12-21T16:01:30.813 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T16:00:54.978785+0000 '>' 2019-12-21T16:00:54.978785+0000 2019-12-21T16:01:30.814 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T16:01:31.814 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T16:01:31.814 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T16:01:31.815 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2019-12-21T16:01:31.815 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.0 2019-12-21T16:01:31.815 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T16:01:31.816 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T16:01:31.816 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2019-12-21T16:01:32.048 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T16:00:54.978785+0000 '>' 2019-12-21T16:00:54.978785+0000 2019-12-21T16:01:32.048 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T16:01:33.050 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T16:01:33.050 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T16:01:33.050 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2019-12-21T16:01:33.050 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.0 2019-12-21T16:01:33.050 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T16:01:33.050 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T16:01:33.051 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2019-12-21T16:01:33.281 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T16:00:54.978785+0000 '>' 2019-12-21T16:00:54.978785+0000 2019-12-21T16:01:33.281 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T16:01:34.282 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T16:01:34.282 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T16:01:34.283 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2019-12-21T16:01:34.283 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.0 2019-12-21T16:01:34.284 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T16:01:34.284 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T16:01:34.284 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2019-12-21T16:01:34.508 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T16:00:54.978785+0000 '>' 2019-12-21T16:00:54.978785+0000 2019-12-21T16:01:34.508 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T16:01:35.509 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T16:01:35.509 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T16:01:35.510 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2019-12-21T16:01:35.510 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.0 2019-12-21T16:01:35.510 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T16:01:35.510 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T16:01:35.510 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2019-12-21T16:01:35.743 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T16:01:33.538102+0000 '>' 2019-12-21T16:00:54.978785+0000 2019-12-21T16:01:35.743 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1795: wait_for_scrub: return 0 2019-12-21T16:01:35.743 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:549: TEST_auto_repair_bluestore_failed_norecov: wait_for_clean 2019-12-21T16:01:35.743 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1548: wait_for_clean: local cmd= 2019-12-21T16:01:35.743 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1549: wait_for_clean: local num_active_clean=-1 2019-12-21T16:01:35.743 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1550: wait_for_clean: local cur_active_clean 2019-12-21T16:01:35.744 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1551: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2019-12-21T16:01:35.744 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1551: wait_for_clean: get_timeout_delays 90 .1 2019-12-21T16:01:35.744 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1494: get_timeout_delays: shopt -q -o xtrace 2019-12-21T16:01:35.744 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1494: get_timeout_delays: echo true 2019-12-21T16:01:35.744 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1494: get_timeout_delays: local trace=true 2019-12-21T16:01:35.744 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1495: get_timeout_delays: true 2019-12-21T16:01:35.744 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1495: get_timeout_delays: shopt -u -o xtrace 2019-12-21T16:01:35.835 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1551: wait_for_clean: local -a delays 2019-12-21T16:01:35.835 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1552: wait_for_clean: local -i loop=0 2019-12-21T16:01:35.836 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1554: wait_for_clean: flush_pg_stats 2019-12-21T16:01:35.836 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1977: flush_pg_stats: local timeout=300 2019-12-21T16:01:35.836 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1979: flush_pg_stats: ceph osd ls 2019-12-21T16:01:36.062 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1979: flush_pg_stats: ids='0 2019-12-21T16:01:36.062 INFO:tasks.workunit.client.0.smithi172.stderr:1 2019-12-21T16:01:36.063 INFO:tasks.workunit.client.0.smithi172.stderr:2' 2019-12-21T16:01:36.063 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1980: flush_pg_stats: seqs= 2019-12-21T16:01:36.063 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: flush_pg_stats: for osd in $ids 2019-12-21T16:01:36.064 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2019-12-21T16:01:36.174 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: seq=137438953479 2019-12-21T16:01:36.174 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: flush_pg_stats: seqs=' 0-137438953479' 2019-12-21T16:01:36.174 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: flush_pg_stats: for osd in $ids 2019-12-21T16:01:36.174 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2019-12-21T16:01:36.285 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: seq=154618822661 2019-12-21T16:01:36.285 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: flush_pg_stats: seqs=' 0-137438953479 1-154618822661' 2019-12-21T16:01:36.285 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: flush_pg_stats: for osd in $ids 2019-12-21T16:01:36.285 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2019-12-21T16:01:36.395 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: seq=64424509455 2019-12-21T16:01:36.396 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: flush_pg_stats: seqs=' 0-137438953479 1-154618822661 2-64424509455' 2019-12-21T16:01:36.396 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: flush_pg_stats: for s in $seqs 2019-12-21T16:01:36.396 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: echo 0-137438953479 2019-12-21T16:01:36.396 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: cut -d - -f 1 2019-12-21T16:01:36.397 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: osd=0 2019-12-21T16:01:36.397 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: echo 0-137438953479 2019-12-21T16:01:36.397 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: cut -d - -f 2 2019-12-21T16:01:36.398 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: seq=137438953479 2019-12-21T16:01:36.398 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1989: flush_pg_stats: echo 'waiting osd.0 seq 137438953479' 2019-12-21T16:01:36.398 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 0 2019-12-21T16:01:36.399 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.0 seq 137438953479 2019-12-21T16:01:36.626 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 137438953478 -lt 137438953479 2019-12-21T16:01:36.626 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1991: flush_pg_stats: sleep 1 2019-12-21T16:01:37.627 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1992: flush_pg_stats: '[' 300 -eq 0 ']' 2019-12-21T16:01:37.627 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 0 2019-12-21T16:01:37.866 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 137438953480 -lt 137438953479 2019-12-21T16:01:37.867 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: flush_pg_stats: for s in $seqs 2019-12-21T16:01:37.867 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: echo 1-154618822661 2019-12-21T16:01:37.867 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: cut -d - -f 1 2019-12-21T16:01:37.868 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: osd=1 2019-12-21T16:01:37.868 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: echo 1-154618822661 2019-12-21T16:01:37.868 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: cut -d - -f 2 2019-12-21T16:01:37.869 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: seq=154618822661 2019-12-21T16:01:37.870 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1989: flush_pg_stats: echo 'waiting osd.1 seq 154618822661' 2019-12-21T16:01:37.870 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 1 2019-12-21T16:01:37.870 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.1 seq 154618822661 2019-12-21T16:01:38.096 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 154618822661 -lt 154618822661 2019-12-21T16:01:38.096 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: flush_pg_stats: for s in $seqs 2019-12-21T16:01:38.097 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: echo 2-64424509455 2019-12-21T16:01:38.097 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: cut -d - -f 1 2019-12-21T16:01:38.097 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: osd=2 2019-12-21T16:01:38.098 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: echo 2-64424509455 2019-12-21T16:01:38.098 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: cut -d - -f 2 2019-12-21T16:01:38.099 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: seq=64424509455 2019-12-21T16:01:38.099 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1989: flush_pg_stats: echo 'waiting osd.2 seq 64424509455' 2019-12-21T16:01:38.099 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 2 2019-12-21T16:01:38.099 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.2 seq 64424509455 2019-12-21T16:01:38.328 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 64424509455 -lt 64424509455 2019-12-21T16:01:38.328 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1555: wait_for_clean: get_num_pgs 2019-12-21T16:01:38.329 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: ceph --format json status 2019-12-21T16:01:38.329 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: jq .pgmap.num_pgs 2019-12-21T16:01:38.601 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1555: wait_for_clean: test 1 == 0 2019-12-21T16:01:38.601 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1559: wait_for_clean: true 2019-12-21T16:01:38.601 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1563: wait_for_clean: get_num_active_clean 2019-12-21T16:01:38.601 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: get_num_active_clean: local expression 2019-12-21T16:01:38.601 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2019-12-21T16:01:38.601 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("stale") | not)' 2019-12-21T16:01:38.602 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: ceph --format json pg dump pgs 2019-12-21T16:01:38.602 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2019-12-21T16:01:38.822 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1563: wait_for_clean: cur_active_clean=1 2019-12-21T16:01:38.823 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1564: wait_for_clean: get_num_pgs 2019-12-21T16:01:38.823 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: ceph --format json status 2019-12-21T16:01:38.824 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: jq .pgmap.num_pgs 2019-12-21T16:01:39.091 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1564: wait_for_clean: test 1 = 1 2019-12-21T16:01:39.092 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1564: wait_for_clean: break 2019-12-21T16:01:39.092 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: return 0 2019-12-21T16:01:39.092 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:550: TEST_auto_repair_bluestore_failed_norecov: flush_pg_stats 2019-12-21T16:01:39.092 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1977: flush_pg_stats: local timeout=300 2019-12-21T16:01:39.092 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1979: flush_pg_stats: ceph osd ls 2019-12-21T16:01:39.320 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1979: flush_pg_stats: ids='0 2019-12-21T16:01:39.320 INFO:tasks.workunit.client.0.smithi172.stderr:1 2019-12-21T16:01:39.320 INFO:tasks.workunit.client.0.smithi172.stderr:2' 2019-12-21T16:01:39.320 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1980: flush_pg_stats: seqs= 2019-12-21T16:01:39.321 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: flush_pg_stats: for osd in $ids 2019-12-21T16:01:39.321 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2019-12-21T16:01:39.431 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: seq=137438953481 2019-12-21T16:01:39.431 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: flush_pg_stats: seqs=' 0-137438953481' 2019-12-21T16:01:39.431 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: flush_pg_stats: for osd in $ids 2019-12-21T16:01:39.431 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2019-12-21T16:01:39.541 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: seq=154618822662 2019-12-21T16:01:39.541 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: flush_pg_stats: seqs=' 0-137438953481 1-154618822662' 2019-12-21T16:01:39.541 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: flush_pg_stats: for osd in $ids 2019-12-21T16:01:39.541 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2019-12-21T16:01:39.651 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: seq=64424509457 2019-12-21T16:01:39.651 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: flush_pg_stats: seqs=' 0-137438953481 1-154618822662 2-64424509457' 2019-12-21T16:01:39.652 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: flush_pg_stats: for s in $seqs 2019-12-21T16:01:39.652 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: echo 0-137438953481 2019-12-21T16:01:39.652 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: cut -d - -f 1 2019-12-21T16:01:39.653 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: osd=0 2019-12-21T16:01:39.653 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: echo 0-137438953481 2019-12-21T16:01:39.654 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: cut -d - -f 2 2019-12-21T16:01:39.654 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: seq=137438953481 2019-12-21T16:01:39.654 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1989: flush_pg_stats: echo 'waiting osd.0 seq 137438953481' 2019-12-21T16:01:39.655 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 0 2019-12-21T16:01:39.655 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.0 seq 137438953481 2019-12-21T16:01:39.895 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 137438953481 -lt 137438953481 2019-12-21T16:01:39.895 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: flush_pg_stats: for s in $seqs 2019-12-21T16:01:39.896 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: echo 1-154618822662 2019-12-21T16:01:39.896 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: cut -d - -f 1 2019-12-21T16:01:39.896 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: osd=1 2019-12-21T16:01:39.897 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: echo 1-154618822662 2019-12-21T16:01:39.897 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: cut -d - -f 2 2019-12-21T16:01:39.898 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: seq=154618822662 2019-12-21T16:01:39.898 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1989: flush_pg_stats: echo 'waiting osd.1 seq 154618822662' 2019-12-21T16:01:39.898 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.1 seq 154618822662 2019-12-21T16:01:39.898 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 1 2019-12-21T16:01:40.126 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 154618822663 -lt 154618822662 2019-12-21T16:01:40.126 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: flush_pg_stats: for s in $seqs 2019-12-21T16:01:40.127 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: echo 2-64424509457 2019-12-21T16:01:40.127 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: cut -d - -f 1 2019-12-21T16:01:40.127 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: osd=2 2019-12-21T16:01:40.128 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: echo 2-64424509457 2019-12-21T16:01:40.128 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: cut -d - -f 2 2019-12-21T16:01:40.129 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: seq=64424509457 2019-12-21T16:01:40.129 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1989: flush_pg_stats: echo 'waiting osd.2 seq 64424509457' 2019-12-21T16:01:40.129 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 2 2019-12-21T16:01:40.129 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.2 seq 64424509457 2019-12-21T16:01:40.358 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 64424509457 -lt 64424509457 2019-12-21T16:01:40.358 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:551: TEST_auto_repair_bluestore_failed_norecov: grep -q 'scrub_finish.*present with no repair possible' td/osd-scrub-repair/osd.1.log 2019-12-21T16:01:40.360 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:552: TEST_auto_repair_bluestore_failed_norecov: ceph pg dump pgs 2019-12-21T16:01:40.572 INFO:tasks.workunit.client.0.smithi172.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2019-12-21T16:01:40.572 INFO:tasks.workunit.client.0.smithi172.stdout:1.0 10 0 0 0 0 56 0 0 10 10 active+clean+inconsistent+failed_repair 2019-12-21T16:01:33.538137+0000 21'10 37:87 [1,0] 1 [1,0] 1 21'10 2019-12-21T16:01:33.538102+0000 21'10 2019-12-21T16:01:33.538102+0000 0 2019-12-21T16:01:40.572 INFO:tasks.workunit.client.0.smithi172.stdout: 2019-12-21T16:01:40.572 INFO:tasks.workunit.client.0.smithi172.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilisation. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2019-12-21T16:01:40.573 INFO:tasks.workunit.client.0.smithi172.stderr:dumped pgs 2019-12-21T16:01:40.582 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:553: TEST_auto_repair_bluestore_failed_norecov: ceph pg dump pgs 2019-12-21T16:01:40.582 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:553: TEST_auto_repair_bluestore_failed_norecov: pgid 2019-12-21T16:01:40.582 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh: line 553: pgid: command not found 2019-12-21T16:01:40.583 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:553: TEST_auto_repair_bluestore_failed_norecov: grep -q '^.*+failed_repair' 2019-12-21T16:01:40.795 INFO:tasks.workunit.client.0.smithi172.stderr:dumped pgs 2019-12-21T16:01:40.803 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:556: TEST_auto_repair_bluestore_failed_norecov: teardown td/osd-scrub-repair 2019-12-21T16:01:40.804 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2019-12-21T16:01:40.804 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2019-12-21T16:01:40.804 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2019-12-21T16:01:40.804 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2019-12-21T16:01:40.804 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2019-12-21T16:01:40.804 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2019-12-21T16:01:40.805 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2019-12-21T16:01:40.805 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2019-12-21T16:01:40.910 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2019-12-21T16:01:40.911 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2019-12-21T16:01:40.911 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2019-12-21T16:01:40.911 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2019-12-21T16:01:40.912 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2019-12-21T16:01:40.912 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2019-12-21T16:01:40.913 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2019-12-21T16:01:40.914 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local 'pattern=|/usr/lib/systemd/systemd-coredump %P %u %g %s %t %c %h %e' 2019-12-21T16:01:40.914 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' '|' = '|' ']' 2019-12-21T16:01:40.914 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:177: teardown: pattern= 2019-12-21T16:01:40.915 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2019-12-21T16:01:40.915 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname '' 2019-12-21T16:01:40.915 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls . 2019-12-21T16:01:40.916 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2019-12-21T16:01:40.916 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2019-12-21T16:01:40.923 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2019-12-21T16:01:40.923 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T16:01:40.923 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.119575 2019-12-21T16:01:40.923 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.119575 2019-12-21T16:01:40.925 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2019-12-21T16:01:40.925 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2019-12-21T16:01:40.925 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:62: run: for func in $funcs 2019-12-21T16:01:40.925 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: TEST_auto_repair_bluestore_scrub td/osd-scrub-repair 2019-12-21T16:01:40.926 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:382: TEST_auto_repair_bluestore_scrub: local dir=td/osd-scrub-repair 2019-12-21T16:01:40.926 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:383: TEST_auto_repair_bluestore_scrub: local poolname=testpool 2019-12-21T16:01:40.926 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:386: TEST_auto_repair_bluestore_scrub: setup td/osd-scrub-repair 2019-12-21T16:01:40.926 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2019-12-21T16:01:40.926 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2019-12-21T16:01:40.926 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2019-12-21T16:01:40.926 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2019-12-21T16:01:40.927 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2019-12-21T16:01:40.927 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2019-12-21T16:01:40.927 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2019-12-21T16:01:40.927 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2019-12-21T16:01:40.927 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2019-12-21T16:01:40.927 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2019-12-21T16:01:40.928 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2019-12-21T16:01:40.928 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2019-12-21T16:01:40.929 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2019-12-21T16:01:40.929 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2019-12-21T16:01:40.929 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2019-12-21T16:01:40.929 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2019-12-21T16:01:40.929 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2019-12-21T16:01:40.930 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local 'pattern=|/usr/lib/systemd/systemd-coredump %P %u %g %s %t %c %h %e' 2019-12-21T16:01:40.930 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' '|' = '|' ']' 2019-12-21T16:01:40.930 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:177: teardown: pattern= 2019-12-21T16:01:40.930 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2019-12-21T16:01:40.931 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname '' 2019-12-21T16:01:40.931 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls . 2019-12-21T16:01:40.932 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2019-12-21T16:01:40.932 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2019-12-21T16:01:40.933 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2019-12-21T16:01:40.933 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T16:01:40.933 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.119575 2019-12-21T16:01:40.933 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.119575 2019-12-21T16:01:40.934 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2019-12-21T16:01:40.934 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2019-12-21T16:01:40.934 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2019-12-21T16:01:40.935 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2019-12-21T16:01:40.935 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T16:01:40.935 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.119575 2019-12-21T16:01:40.936 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.119575 2019-12-21T16:01:40.937 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2019-12-21T16:01:40.937 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2019-12-21T16:01:40.938 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2019-12-21T16:01:40.938 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2019-12-21T16:01:40.938 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:387: TEST_auto_repair_bluestore_scrub: run_mon td/osd-scrub-repair a 2019-12-21T16:01:40.938 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2019-12-21T16:01:40.938 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2019-12-21T16:01:40.938 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2019-12-21T16:01:40.938 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2019-12-21T16:01:40.939 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2019-12-21T16:01:40.939 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2019-12-21T16:01:41.231 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2019-12-21T16:01:41.232 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2019-12-21T16:01:41.232 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2019-12-21T16:01:41.232 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2019-12-21T16:01:41.232 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T16:01:41.232 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.119575 2019-12-21T16:01:41.232 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.119575/$cluster-$name.asok' 2019-12-21T16:01:41.233 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.119575/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 2019-12-21T16:01:41.474 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: cat 2019-12-21T16:01:41.476 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: get_config mon a fsid 2019-12-21T16:01:41.476 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: get_config: local daemon=mon 2019-12-21T16:01:41.476 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: get_config: local id=a 2019-12-21T16:01:41.476 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local config=fsid 2019-12-21T16:01:41.476 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: jq -r .fsid 2019-12-21T16:01:41.476 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: get_asok_path mon.a 2019-12-21T16:01:41.476 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2019-12-21T16:01:41.477 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2019-12-21T16:01:41.477 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T16:01:41.477 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T16:01:41.477 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.119575 2019-12-21T16:01:41.477 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.119575/ceph-mon.a.asok 2019-12-21T16:01:41.478 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: CEPH_ARGS= 2019-12-21T16:01:41.478 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: ceph --format json daemon /tmp/ceph-asok.119575/ceph-mon.a.asok config get fsid 2019-12-21T16:01:41.564 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: get_config mon a mon_host 2019-12-21T16:01:41.564 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: get_config: local daemon=mon 2019-12-21T16:01:41.564 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: get_config: local id=a 2019-12-21T16:01:41.564 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local config=mon_host 2019-12-21T16:01:41.565 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: jq -r .mon_host 2019-12-21T16:01:41.565 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: get_asok_path mon.a 2019-12-21T16:01:41.565 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2019-12-21T16:01:41.565 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2019-12-21T16:01:41.565 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T16:01:41.565 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T16:01:41.566 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.119575 2019-12-21T16:01:41.566 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.119575/ceph-mon.a.asok 2019-12-21T16:01:41.566 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: CEPH_ARGS= 2019-12-21T16:01:41.566 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: ceph --format json daemon /tmp/ceph-asok.119575/ceph-mon.a.asok config get mon_host 2019-12-21T16:01:41.650 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:388: TEST_auto_repair_bluestore_scrub: run_mgr td/osd-scrub-repair x 2019-12-21T16:01:41.651 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:551: run_mgr: local dir=td/osd-scrub-repair 2019-12-21T16:01:41.651 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: shift 2019-12-21T16:01:41.651 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local id=x 2019-12-21T16:01:41.651 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2019-12-21T16:01:41.651 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local data=td/osd-scrub-repair/x 2019-12-21T16:01:41.651 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:558: run_mgr: get_asok_path 2019-12-21T16:01:41.652 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2019-12-21T16:01:41.652 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2019-12-21T16:01:41.652 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2019-12-21T16:01:41.652 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T16:01:41.652 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.119575 2019-12-21T16:01:41.652 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.119575/$cluster-$name.asok' 2019-12-21T16:01:41.653 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:558: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2019-12-21T16:01:41.653 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:558: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.119575/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2019-12-21T16:01:42.273 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:391: TEST_auto_repair_bluestore_scrub: local 'ceph_osd_args=--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2019-12-21T16:01:42.274 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:392: TEST_auto_repair_bluestore_scrub: seq 0 2 2019-12-21T16:01:42.280 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:392: TEST_auto_repair_bluestore_scrub: for id in $(seq 0 2) 2019-12-21T16:01:42.280 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:393: TEST_auto_repair_bluestore_scrub: run_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2019-12-21T16:01:42.280 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local dir=td/osd-scrub-repair 2019-12-21T16:01:42.281 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: shift 2019-12-21T16:01:42.281 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local id=0 2019-12-21T16:01:42.281 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2019-12-21T16:01:42.281 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local osd_data=td/osd-scrub-repair/0 2019-12-21T16:01:42.281 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local 'ceph_args=--fsid=973e3fc1-e63e-4bde-9db6-84c68fd5e93e --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2019-12-21T16:01:42.282 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2019-12-21T16:01:42.282 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-journal-size=100' 2019-12-21T16:01:42.282 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2019-12-21T16:01:42.282 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2019-12-21T16:01:42.282 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2019-12-21T16:01:42.282 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --chdir=' 2019-12-21T16:01:42.283 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+= 2019-12-21T16:01:42.283 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2019-12-21T16:01:42.283 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: get_asok_path 2019-12-21T16:01:42.283 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2019-12-21T16:01:42.283 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2019-12-21T16:01:42.283 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2019-12-21T16:01:42.283 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T16:01:42.284 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.119575 2019-12-21T16:01:42.284 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.119575/$cluster-$name.asok' 2019-12-21T16:01:42.284 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.119575/$cluster-$name.asok' 2019-12-21T16:01:42.284 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --debug-osd=20' 2019-12-21T16:01:42.284 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-ms=1' 2019-12-21T16:01:42.284 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-monc=20' 2019-12-21T16:01:42.285 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2019-12-21T16:01:42.285 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2019-12-21T16:01:42.285 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2019-12-21T16:01:42.285 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2019-12-21T16:01:42.285 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2019-12-21T16:01:42.285 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' ' 2019-12-21T16:01:42.285 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2019-12-21T16:01:42.286 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: mkdir -p td/osd-scrub-repair/0 2019-12-21T16:01:42.286 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: uuidgen 2019-12-21T16:01:42.286 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: local uuid=ab6e82fb-01d4-4001-a78c-831e1be0fa3c 2019-12-21T16:01:42.286 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: echo 'add osd0 ab6e82fb-01d4-4001-a78c-831e1be0fa3c' 2019-12-21T16:01:42.286 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: ceph-authtool --gen-print-key 2019-12-21T16:01:42.286 INFO:tasks.workunit.client.0.smithi172.stdout:add osd0 ab6e82fb-01d4-4001-a78c-831e1be0fa3c 2019-12-21T16:01:42.292 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: OSD_SECRET=AQDmQf5dZ29DERAA4d/acCR2hYWieqy8wqEsQA== 2019-12-21T16:01:42.292 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo '{"cephx_secret": "AQDmQf5dZ29DERAA4d/acCR2hYWieqy8wqEsQA=="}' 2019-12-21T16:01:42.292 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph osd new ab6e82fb-01d4-4001-a78c-831e1be0fa3c -i td/osd-scrub-repair/0/new.json 2019-12-21T16:01:42.540 INFO:tasks.workunit.client.0.smithi172.stdout:0 2019-12-21T16:01:42.550 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: rm td/osd-scrub-repair/0/new.json 2019-12-21T16:01:42.550 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-osd -i 0 --fsid=973e3fc1-e63e-4bde-9db6-84c68fd5e93e --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.119575/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQDmQf5dZ29DERAA4d/acCR2hYWieqy8wqEsQA== --osd-uuid ab6e82fb-01d4-4001-a78c-831e1be0fa3c 2019-12-21T16:01:42.573 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:01:42.572+0000 7f63b943aec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T16:01:42.575 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:01:42.574+0000 7f63b943aec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T16:01:42.577 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:01:42.575+0000 7f63b943aec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T16:01:42.578 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:01:42.576+0000 7f63b943aec0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2019-12-21T16:01:42.578 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:01:42.576+0000 7f63b943aec0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2019-12-21T16:01:42.578 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:01:42.576+0000 7f63b943aec0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2019-12-21T16:01:42.631 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:01:42.630+0000 7f63b943aec0 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2019-12-21T16:01:45.601 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2019-12-21T16:01:45.601 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: cat 2019-12-21T16:01:45.602 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: echo adding osd0 key to auth repository 2019-12-21T16:01:45.602 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2019-12-21T16:01:45.603 INFO:tasks.workunit.client.0.smithi172.stdout:adding osd0 key to auth repository 2019-12-21T16:01:45.781 INFO:tasks.workunit.client.0.smithi172.stdout:start osd.0 2019-12-21T16:01:45.782 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo start osd.0 2019-12-21T16:01:45.782 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph-osd -i 0 --fsid=973e3fc1-e63e-4bde-9db6-84c68fd5e93e --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.119575/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2019-12-21T16:01:45.782 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph osd dump --format=json 2019-12-21T16:01:45.782 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: jq '.flags_set[]' 2019-12-21T16:01:45.782 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: grep -q '"noup"' 2019-12-21T16:01:45.802 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:01:45.801+0000 7f4152b01ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T16:01:45.824 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:01:45.822+0000 7f4152b01ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T16:01:45.825 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:01:45.823+0000 7f4152b01ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T16:01:46.011 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: wait_for_osd up 0 2019-12-21T16:01:46.011 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:914: wait_for_osd: local state=up 2019-12-21T16:01:46.011 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local id=0 2019-12-21T16:01:46.012 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: status=1 2019-12-21T16:01:46.012 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i=0 )) 2019-12-21T16:01:46.012 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i < 300 )) 2019-12-21T16:01:46.012 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: echo 0 2019-12-21T16:01:46.012 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: ceph osd dump 2019-12-21T16:01:46.012 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: grep 'osd.0 up' 2019-12-21T16:01:46.013 INFO:tasks.workunit.client.0.smithi172.stdout:0 2019-12-21T16:01:46.239 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: sleep 1 2019-12-21T16:01:46.340 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:01:46.338+0000 7f4152b01ec0 -1 Falling back to public interface 2019-12-21T16:01:47.030 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:01:47.027+0000 7f4152b01ec0 -1 osd.0 0 log_to_monitors {default=true} 2019-12-21T16:01:47.240 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i++ )) 2019-12-21T16:01:47.241 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i < 300 )) 2019-12-21T16:01:47.241 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: echo 1 2019-12-21T16:01:47.241 INFO:tasks.workunit.client.0.smithi172.stdout:1 2019-12-21T16:01:47.242 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: grep 'osd.0 up' 2019-12-21T16:01:47.242 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: ceph osd dump 2019-12-21T16:01:47.511 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: sleep 1 2019-12-21T16:01:48.512 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i++ )) 2019-12-21T16:01:48.513 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i < 300 )) 2019-12-21T16:01:48.513 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: echo 2 2019-12-21T16:01:48.513 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: ceph osd dump 2019-12-21T16:01:48.513 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: grep 'osd.0 up' 2019-12-21T16:01:48.513 INFO:tasks.workunit.client.0.smithi172.stdout:2 2019-12-21T16:01:48.751 INFO:tasks.workunit.client.0.smithi172.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/140994,v1:127.0.0.1:6803/140994] [v2:127.0.0.1:6804/140994,v1:127.0.0.1:6805/140994] exists,up ab6e82fb-01d4-4001-a78c-831e1be0fa3c 2019-12-21T16:01:48.752 INFO:tasks.workunit.client.0.smithi172.stdout:add osd1 fd8b19dc-60be-46fc-aed4-78f8d0fd59cd 2019-12-21T16:01:48.752 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=0 2019-12-21T16:01:48.752 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: break 2019-12-21T16:01:48.752 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: return 0 2019-12-21T16:01:48.752 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:392: TEST_auto_repair_bluestore_scrub: for id in $(seq 0 2) 2019-12-21T16:01:48.752 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:393: TEST_auto_repair_bluestore_scrub: run_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2019-12-21T16:01:48.753 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local dir=td/osd-scrub-repair 2019-12-21T16:01:48.753 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: shift 2019-12-21T16:01:48.753 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local id=1 2019-12-21T16:01:48.753 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2019-12-21T16:01:48.753 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local osd_data=td/osd-scrub-repair/1 2019-12-21T16:01:48.753 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local 'ceph_args=--fsid=973e3fc1-e63e-4bde-9db6-84c68fd5e93e --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2019-12-21T16:01:48.753 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2019-12-21T16:01:48.754 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-journal-size=100' 2019-12-21T16:01:48.754 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2019-12-21T16:01:48.754 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2019-12-21T16:01:48.754 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2019-12-21T16:01:48.754 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --chdir=' 2019-12-21T16:01:48.754 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+= 2019-12-21T16:01:48.754 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2019-12-21T16:01:48.755 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: get_asok_path 2019-12-21T16:01:48.755 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2019-12-21T16:01:48.755 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2019-12-21T16:01:48.755 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2019-12-21T16:01:48.755 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T16:01:48.755 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.119575 2019-12-21T16:01:48.755 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.119575/$cluster-$name.asok' 2019-12-21T16:01:48.756 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.119575/$cluster-$name.asok' 2019-12-21T16:01:48.756 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --debug-osd=20' 2019-12-21T16:01:48.756 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-ms=1' 2019-12-21T16:01:48.756 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-monc=20' 2019-12-21T16:01:48.756 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2019-12-21T16:01:48.756 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2019-12-21T16:01:48.756 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2019-12-21T16:01:48.757 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2019-12-21T16:01:48.757 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2019-12-21T16:01:48.757 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' ' 2019-12-21T16:01:48.757 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2019-12-21T16:01:48.757 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: mkdir -p td/osd-scrub-repair/1 2019-12-21T16:01:48.757 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: uuidgen 2019-12-21T16:01:48.757 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: local uuid=fd8b19dc-60be-46fc-aed4-78f8d0fd59cd 2019-12-21T16:01:48.758 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: echo 'add osd1 fd8b19dc-60be-46fc-aed4-78f8d0fd59cd' 2019-12-21T16:01:48.758 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: ceph-authtool --gen-print-key 2019-12-21T16:01:48.765 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: OSD_SECRET=AQDsQf5dqOd8LRAAdAXqGPoyWr9HRGuUMFQ7IA== 2019-12-21T16:01:48.765 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo '{"cephx_secret": "AQDsQf5dqOd8LRAAdAXqGPoyWr9HRGuUMFQ7IA=="}' 2019-12-21T16:01:48.765 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph osd new fd8b19dc-60be-46fc-aed4-78f8d0fd59cd -i td/osd-scrub-repair/1/new.json 2019-12-21T16:01:49.071 INFO:tasks.workunit.client.0.smithi172.stdout:1 2019-12-21T16:01:49.080 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: rm td/osd-scrub-repair/1/new.json 2019-12-21T16:01:49.082 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-osd -i 1 --fsid=973e3fc1-e63e-4bde-9db6-84c68fd5e93e --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.119575/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQDsQf5dqOd8LRAAdAXqGPoyWr9HRGuUMFQ7IA== --osd-uuid fd8b19dc-60be-46fc-aed4-78f8d0fd59cd 2019-12-21T16:01:49.104 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:01:49.102+0000 7f0ac25abec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T16:01:49.106 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:01:49.103+0000 7f0ac25abec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T16:01:49.107 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:01:49.104+0000 7f0ac25abec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T16:01:49.107 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:01:49.105+0000 7f0ac25abec0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2019-12-21T16:01:49.108 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:01:49.105+0000 7f0ac25abec0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2019-12-21T16:01:49.108 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:01:49.105+0000 7f0ac25abec0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2019-12-21T16:01:49.154 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:01:49.152+0000 7f0ac25abec0 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2019-12-21T16:01:52.197 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2019-12-21T16:01:52.197 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: cat 2019-12-21T16:01:52.198 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: echo adding osd1 key to auth repository 2019-12-21T16:01:52.198 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2019-12-21T16:01:52.198 INFO:tasks.workunit.client.0.smithi172.stdout:adding osd1 key to auth repository 2019-12-21T16:01:52.482 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo start osd.1 2019-12-21T16:01:52.482 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph-osd -i 1 --fsid=973e3fc1-e63e-4bde-9db6-84c68fd5e93e --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.119575/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2019-12-21T16:01:52.482 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph osd dump --format=json 2019-12-21T16:01:52.482 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: jq '.flags_set[]' 2019-12-21T16:01:52.482 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: grep -q '"noup"' 2019-12-21T16:01:52.482 INFO:tasks.workunit.client.0.smithi172.stdout:start osd.1 2019-12-21T16:01:52.503 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:01:52.500+0000 7f921b6c7ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T16:01:52.504 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:01:52.502+0000 7f921b6c7ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T16:01:52.505 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:01:52.503+0000 7f921b6c7ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T16:01:52.711 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: wait_for_osd up 1 2019-12-21T16:01:52.711 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:914: wait_for_osd: local state=up 2019-12-21T16:01:52.711 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local id=1 2019-12-21T16:01:52.711 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: status=1 2019-12-21T16:01:52.711 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i=0 )) 2019-12-21T16:01:52.712 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i < 300 )) 2019-12-21T16:01:52.712 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: echo 0 2019-12-21T16:01:52.712 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: ceph osd dump 2019-12-21T16:01:52.712 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: grep 'osd.1 up' 2019-12-21T16:01:52.712 INFO:tasks.workunit.client.0.smithi172.stdout:0 2019-12-21T16:01:52.938 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: sleep 1 2019-12-21T16:01:53.021 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:01:53.019+0000 7f921b6c7ec0 -1 Falling back to public interface 2019-12-21T16:01:53.837 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:01:53.835+0000 7f921b6c7ec0 -1 osd.1 0 log_to_monitors {default=true} 2019-12-21T16:01:53.939 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i++ )) 2019-12-21T16:01:53.940 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i < 300 )) 2019-12-21T16:01:53.940 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: echo 1 2019-12-21T16:01:53.940 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: ceph osd dump 2019-12-21T16:01:53.940 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: grep 'osd.1 up' 2019-12-21T16:01:53.940 INFO:tasks.workunit.client.0.smithi172.stdout:1 2019-12-21T16:01:54.179 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: sleep 1 2019-12-21T16:01:55.180 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i++ )) 2019-12-21T16:01:55.180 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i < 300 )) 2019-12-21T16:01:55.181 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: echo 2 2019-12-21T16:01:55.181 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: ceph osd dump 2019-12-21T16:01:55.181 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: grep 'osd.1 up' 2019-12-21T16:01:55.181 INFO:tasks.workunit.client.0.smithi172.stdout:2 2019-12-21T16:01:55.421 INFO:tasks.workunit.client.0.smithi172.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/141371,v1:127.0.0.1:6811/141371] [v2:127.0.0.1:6812/141371,v1:127.0.0.1:6813/141371] exists,up fd8b19dc-60be-46fc-aed4-78f8d0fd59cd 2019-12-21T16:01:55.422 INFO:tasks.workunit.client.0.smithi172.stdout:add osd2 f9bc4938-afed-4fb2-a29e-503c0670ec25 2019-12-21T16:01:55.422 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=0 2019-12-21T16:01:55.422 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: break 2019-12-21T16:01:55.422 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: return 0 2019-12-21T16:01:55.422 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:392: TEST_auto_repair_bluestore_scrub: for id in $(seq 0 2) 2019-12-21T16:01:55.422 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:393: TEST_auto_repair_bluestore_scrub: run_osd td/osd-scrub-repair 2 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2019-12-21T16:01:55.423 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local dir=td/osd-scrub-repair 2019-12-21T16:01:55.423 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: shift 2019-12-21T16:01:55.423 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local id=2 2019-12-21T16:01:55.423 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2019-12-21T16:01:55.423 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local osd_data=td/osd-scrub-repair/2 2019-12-21T16:01:55.423 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local 'ceph_args=--fsid=973e3fc1-e63e-4bde-9db6-84c68fd5e93e --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2019-12-21T16:01:55.424 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2019-12-21T16:01:55.424 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-journal-size=100' 2019-12-21T16:01:55.424 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2019-12-21T16:01:55.424 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2019-12-21T16:01:55.424 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2019-12-21T16:01:55.424 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --chdir=' 2019-12-21T16:01:55.424 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+= 2019-12-21T16:01:55.425 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2019-12-21T16:01:55.425 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: get_asok_path 2019-12-21T16:01:55.425 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2019-12-21T16:01:55.425 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2019-12-21T16:01:55.425 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2019-12-21T16:01:55.425 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T16:01:55.426 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.119575 2019-12-21T16:01:55.426 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.119575/$cluster-$name.asok' 2019-12-21T16:01:55.426 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.119575/$cluster-$name.asok' 2019-12-21T16:01:55.426 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --debug-osd=20' 2019-12-21T16:01:55.426 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-ms=1' 2019-12-21T16:01:55.426 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-monc=20' 2019-12-21T16:01:55.426 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2019-12-21T16:01:55.427 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2019-12-21T16:01:55.427 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2019-12-21T16:01:55.427 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2019-12-21T16:01:55.427 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2019-12-21T16:01:55.427 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' ' 2019-12-21T16:01:55.427 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2019-12-21T16:01:55.428 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: mkdir -p td/osd-scrub-repair/2 2019-12-21T16:01:55.428 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: uuidgen 2019-12-21T16:01:55.428 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: local uuid=f9bc4938-afed-4fb2-a29e-503c0670ec25 2019-12-21T16:01:55.428 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: echo 'add osd2 f9bc4938-afed-4fb2-a29e-503c0670ec25' 2019-12-21T16:01:55.428 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: ceph-authtool --gen-print-key 2019-12-21T16:01:55.434 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: OSD_SECRET=AQDzQf5dgJnBGRAAIEVXXXaZRyTIZl2yakMAYA== 2019-12-21T16:01:55.434 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo '{"cephx_secret": "AQDzQf5dgJnBGRAAIEVXXXaZRyTIZl2yakMAYA=="}' 2019-12-21T16:01:55.434 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph osd new f9bc4938-afed-4fb2-a29e-503c0670ec25 -i td/osd-scrub-repair/2/new.json 2019-12-21T16:01:55.786 INFO:tasks.workunit.client.0.smithi172.stdout:2 2019-12-21T16:01:55.796 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: rm td/osd-scrub-repair/2/new.json 2019-12-21T16:01:55.797 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-osd -i 2 --fsid=973e3fc1-e63e-4bde-9db6-84c68fd5e93e --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.119575/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQDzQf5dgJnBGRAAIEVXXXaZRyTIZl2yakMAYA== --osd-uuid f9bc4938-afed-4fb2-a29e-503c0670ec25 2019-12-21T16:01:55.819 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:01:55.816+0000 7f34e9a03ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T16:01:55.821 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:01:55.818+0000 7f34e9a03ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T16:01:55.823 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:01:55.820+0000 7f34e9a03ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T16:01:55.824 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:01:55.821+0000 7f34e9a03ec0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2019-12-21T16:01:55.824 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:01:55.821+0000 7f34e9a03ec0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2019-12-21T16:01:55.824 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:01:55.821+0000 7f34e9a03ec0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2019-12-21T16:01:55.877 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:01:55.875+0000 7f34e9a03ec0 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2019-12-21T16:01:58.880 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2019-12-21T16:01:58.880 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: cat 2019-12-21T16:01:58.881 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: echo adding osd2 key to auth repository 2019-12-21T16:01:58.881 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2019-12-21T16:01:58.881 INFO:tasks.workunit.client.0.smithi172.stdout:adding osd2 key to auth repository 2019-12-21T16:01:59.163 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo start osd.2 2019-12-21T16:01:59.163 INFO:tasks.workunit.client.0.smithi172.stdout:start osd.2 2019-12-21T16:01:59.164 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph osd dump --format=json 2019-12-21T16:01:59.164 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph-osd -i 2 --fsid=973e3fc1-e63e-4bde-9db6-84c68fd5e93e --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.119575/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2019-12-21T16:01:59.165 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: jq '.flags_set[]' 2019-12-21T16:01:59.165 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: grep -q '"noup"' 2019-12-21T16:01:59.184 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:01:59.182+0000 7fb2e3bb7ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T16:01:59.186 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:01:59.184+0000 7fb2e3bb7ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T16:01:59.188 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:01:59.185+0000 7fb2e3bb7ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T16:01:59.414 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: wait_for_osd up 2 2019-12-21T16:01:59.414 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:914: wait_for_osd: local state=up 2019-12-21T16:01:59.414 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local id=2 2019-12-21T16:01:59.415 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: status=1 2019-12-21T16:01:59.415 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i=0 )) 2019-12-21T16:01:59.415 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i < 300 )) 2019-12-21T16:01:59.415 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: echo 0 2019-12-21T16:01:59.415 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: ceph osd dump 2019-12-21T16:01:59.415 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: grep 'osd.2 up' 2019-12-21T16:01:59.415 INFO:tasks.workunit.client.0.smithi172.stdout:0 2019-12-21T16:01:59.654 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: sleep 1 2019-12-21T16:01:59.707 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:01:59.705+0000 7fb2e3bb7ec0 -1 Falling back to public interface 2019-12-21T16:02:00.435 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:02:00.432+0000 7fb2e3bb7ec0 -1 osd.2 0 log_to_monitors {default=true} 2019-12-21T16:02:00.656 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i++ )) 2019-12-21T16:02:00.656 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i < 300 )) 2019-12-21T16:02:00.656 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: echo 1 2019-12-21T16:02:00.656 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: ceph osd dump 2019-12-21T16:02:00.656 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: grep 'osd.2 up' 2019-12-21T16:02:00.657 INFO:tasks.workunit.client.0.smithi172.stdout:1 2019-12-21T16:02:00.940 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: sleep 1 2019-12-21T16:02:01.942 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i++ )) 2019-12-21T16:02:01.942 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i < 300 )) 2019-12-21T16:02:01.943 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: echo 2 2019-12-21T16:02:01.943 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: ceph osd dump 2019-12-21T16:02:01.943 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: grep 'osd.2 up' 2019-12-21T16:02:01.943 INFO:tasks.workunit.client.0.smithi172.stdout:2 2019-12-21T16:02:02.173 INFO:tasks.workunit.client.0.smithi172.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/141748,v1:127.0.0.1:6819/141748] [v2:127.0.0.1:6820/141748,v1:127.0.0.1:6821/141748] exists,up f9bc4938-afed-4fb2-a29e-503c0670ec25 2019-12-21T16:02:02.173 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=0 2019-12-21T16:02:02.173 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: break 2019-12-21T16:02:02.173 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: return 0 2019-12-21T16:02:02.173 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:396: TEST_auto_repair_bluestore_scrub: create_pool testpool 1 1 2019-12-21T16:02:02.174 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:539: create_pool: ceph osd pool create testpool 1 1 2019-12-21T16:02:02.545 INFO:tasks.workunit.client.0.smithi172.stderr:pool 'testpool' created 2019-12-21T16:02:02.552 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: sleep 1 2019-12-21T16:02:03.554 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:397: TEST_auto_repair_bluestore_scrub: ceph osd pool set testpool size 2 2019-12-21T16:02:03.890 INFO:tasks.workunit.client.0.smithi172.stderr:set pool 1 size to 2 2019-12-21T16:02:03.898 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:398: TEST_auto_repair_bluestore_scrub: wait_for_clean 2019-12-21T16:02:03.898 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1548: wait_for_clean: local cmd= 2019-12-21T16:02:03.898 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1549: wait_for_clean: local num_active_clean=-1 2019-12-21T16:02:03.898 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1550: wait_for_clean: local cur_active_clean 2019-12-21T16:02:03.899 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1551: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2019-12-21T16:02:03.899 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1551: wait_for_clean: get_timeout_delays 90 .1 2019-12-21T16:02:03.899 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1494: get_timeout_delays: shopt -q -o xtrace 2019-12-21T16:02:03.900 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1494: get_timeout_delays: echo true 2019-12-21T16:02:03.900 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1494: get_timeout_delays: local trace=true 2019-12-21T16:02:03.900 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1495: get_timeout_delays: true 2019-12-21T16:02:03.900 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1495: get_timeout_delays: shopt -u -o xtrace 2019-12-21T16:02:03.992 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1551: wait_for_clean: local -a delays 2019-12-21T16:02:03.992 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1552: wait_for_clean: local -i loop=0 2019-12-21T16:02:03.992 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1554: wait_for_clean: flush_pg_stats 2019-12-21T16:02:03.992 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1977: flush_pg_stats: local timeout=300 2019-12-21T16:02:03.993 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1979: flush_pg_stats: ceph osd ls 2019-12-21T16:02:04.221 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1979: flush_pg_stats: ids='0 2019-12-21T16:02:04.221 INFO:tasks.workunit.client.0.smithi172.stderr:1 2019-12-21T16:02:04.222 INFO:tasks.workunit.client.0.smithi172.stderr:2' 2019-12-21T16:02:04.222 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1980: flush_pg_stats: seqs= 2019-12-21T16:02:04.222 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: flush_pg_stats: for osd in $ids 2019-12-21T16:02:04.222 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2019-12-21T16:02:04.332 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: seq=21474836485 2019-12-21T16:02:04.333 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: flush_pg_stats: seqs=' 0-21474836485' 2019-12-21T16:02:04.333 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: flush_pg_stats: for osd in $ids 2019-12-21T16:02:04.333 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2019-12-21T16:02:04.444 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: seq=42949672963 2019-12-21T16:02:04.444 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: flush_pg_stats: seqs=' 0-21474836485 1-42949672963' 2019-12-21T16:02:04.444 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: flush_pg_stats: for osd in $ids 2019-12-21T16:02:04.444 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2019-12-21T16:02:04.554 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: seq=64424509442 2019-12-21T16:02:04.555 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: flush_pg_stats: seqs=' 0-21474836485 1-42949672963 2-64424509442' 2019-12-21T16:02:04.555 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: flush_pg_stats: for s in $seqs 2019-12-21T16:02:04.555 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: echo 0-21474836485 2019-12-21T16:02:04.556 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: cut -d - -f 1 2019-12-21T16:02:04.556 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: osd=0 2019-12-21T16:02:04.556 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: echo 0-21474836485 2019-12-21T16:02:04.557 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: cut -d - -f 2 2019-12-21T16:02:04.557 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: seq=21474836485 2019-12-21T16:02:04.557 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1989: flush_pg_stats: echo 'waiting osd.0 seq 21474836485' 2019-12-21T16:02:04.558 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 0 2019-12-21T16:02:04.558 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.0 seq 21474836485 2019-12-21T16:02:04.786 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 21474836484 -lt 21474836485 2019-12-21T16:02:04.786 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1991: flush_pg_stats: sleep 1 2019-12-21T16:02:05.787 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1992: flush_pg_stats: '[' 300 -eq 0 ']' 2019-12-21T16:02:05.787 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 0 2019-12-21T16:02:06.025 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 21474836485 -lt 21474836485 2019-12-21T16:02:06.025 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: flush_pg_stats: for s in $seqs 2019-12-21T16:02:06.025 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: echo 1-42949672963 2019-12-21T16:02:06.026 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: cut -d - -f 1 2019-12-21T16:02:06.026 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: osd=1 2019-12-21T16:02:06.027 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: echo 1-42949672963 2019-12-21T16:02:06.027 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: cut -d - -f 2 2019-12-21T16:02:06.028 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: seq=42949672963 2019-12-21T16:02:06.028 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1989: flush_pg_stats: echo 'waiting osd.1 seq 42949672963' 2019-12-21T16:02:06.028 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 1 2019-12-21T16:02:06.028 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.1 seq 42949672963 2019-12-21T16:02:06.254 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 42949672964 -lt 42949672963 2019-12-21T16:02:06.255 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: flush_pg_stats: for s in $seqs 2019-12-21T16:02:06.255 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: echo 2-64424509442 2019-12-21T16:02:06.255 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: cut -d - -f 1 2019-12-21T16:02:06.256 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: osd=2 2019-12-21T16:02:06.256 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: echo 2-64424509442 2019-12-21T16:02:06.256 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: cut -d - -f 2 2019-12-21T16:02:06.257 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: seq=64424509442 2019-12-21T16:02:06.258 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1989: flush_pg_stats: echo 'waiting osd.2 seq 64424509442' 2019-12-21T16:02:06.258 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 2 2019-12-21T16:02:06.258 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.2 seq 64424509442 2019-12-21T16:02:06.485 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 64424509442 -lt 64424509442 2019-12-21T16:02:06.486 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1555: wait_for_clean: get_num_pgs 2019-12-21T16:02:06.486 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: ceph --format json status 2019-12-21T16:02:06.486 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: jq .pgmap.num_pgs 2019-12-21T16:02:06.755 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1555: wait_for_clean: test 1 == 0 2019-12-21T16:02:06.755 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1559: wait_for_clean: true 2019-12-21T16:02:06.755 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1563: wait_for_clean: get_num_active_clean 2019-12-21T16:02:06.755 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: get_num_active_clean: local expression 2019-12-21T16:02:06.756 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2019-12-21T16:02:06.756 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("stale") | not)' 2019-12-21T16:02:06.756 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: ceph --format json pg dump pgs 2019-12-21T16:02:06.756 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2019-12-21T16:02:06.978 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1563: wait_for_clean: cur_active_clean=1 2019-12-21T16:02:06.978 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1564: wait_for_clean: get_num_pgs 2019-12-21T16:02:06.979 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: ceph --format json status 2019-12-21T16:02:06.979 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: jq .pgmap.num_pgs 2019-12-21T16:02:07.248 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1564: wait_for_clean: test 1 = 1 2019-12-21T16:02:07.248 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1564: wait_for_clean: break 2019-12-21T16:02:07.248 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: return 0 2019-12-21T16:02:07.248 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:401: TEST_auto_repair_bluestore_scrub: local payload=ABCDEF 2019-12-21T16:02:07.249 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:402: TEST_auto_repair_bluestore_scrub: echo ABCDEF 2019-12-21T16:02:07.249 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:403: TEST_auto_repair_bluestore_scrub: rados --pool testpool put SOMETHING td/osd-scrub-repair/ORIGINAL 2019-12-21T16:02:07.295 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:407: TEST_auto_repair_bluestore_scrub: get_not_primary testpool SOMETHING 2019-12-21T16:02:07.295 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1165: get_not_primary: local poolname=testpool 2019-12-21T16:02:07.295 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1166: get_not_primary: local objectname=SOMETHING 2019-12-21T16:02:07.295 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: get_not_primary: get_primary testpool SOMETHING 2019-12-21T16:02:07.296 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_primary: local poolname=testpool 2019-12-21T16:02:07.296 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1133: get_primary: local objectname=SOMETHING 2019-12-21T16:02:07.296 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1135: get_primary: ceph --format json osd map testpool SOMETHING 2019-12-21T16:02:07.296 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1136: get_primary: jq .acting_primary 2019-12-21T16:02:07.525 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: get_not_primary: local primary=1 2019-12-21T16:02:07.525 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1169: get_not_primary: ceph --format json osd map testpool SOMETHING 2019-12-21T16:02:07.525 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1170: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2019-12-21T16:02:07.759 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:407: TEST_auto_repair_bluestore_scrub: objectstore_tool td/osd-scrub-repair 0 SOMETHING remove 2019-12-21T16:02:07.759 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: objectstore_tool: local dir=td/osd-scrub-repair 2019-12-21T16:02:07.759 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: objectstore_tool: shift 2019-12-21T16:02:07.759 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1235: objectstore_tool: local id=0 2019-12-21T16:02:07.759 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1236: objectstore_tool: shift 2019-12-21T16:02:07.759 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1238: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING remove 2019-12-21T16:02:07.760 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1205: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2019-12-21T16:02:07.760 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nowait: shift 2019-12-21T16:02:07.760 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1207: _objectstore_tool_nowait: local id=0 2019-12-21T16:02:07.760 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1208: _objectstore_tool_nowait: shift 2019-12-21T16:02:07.760 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1210: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2019-12-21T16:02:07.760 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2019-12-21T16:02:07.760 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2019-12-21T16:02:07.761 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2019-12-21T16:02:07.761 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2019-12-21T16:02:07.761 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2019-12-21T16:02:07.864 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2019-12-21T16:02:07.864 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1212: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING remove 2019-12-21T16:02:07.864 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1193: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2019-12-21T16:02:07.864 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1194: _objectstore_tool_nodown: shift 2019-12-21T16:02:07.865 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1195: _objectstore_tool_nodown: local id=0 2019-12-21T16:02:07.865 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: _objectstore_tool_nodown: shift 2019-12-21T16:02:07.865 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2019-12-21T16:02:07.865 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING remove 2019-12-21T16:02:08.494 INFO:tasks.workunit.client.0.smithi172.stdout:remove #1:eb822e21:::SOMETHING:head# 2019-12-21T16:02:09.017 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1213: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2019-12-21T16:02:09.017 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:842: activate_osd: local dir=td/osd-scrub-repair 2019-12-21T16:02:09.017 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: shift 2019-12-21T16:02:09.017 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: local id=0 2019-12-21T16:02:09.018 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: shift 2019-12-21T16:02:09.018 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local osd_data=td/osd-scrub-repair/0 2019-12-21T16:02:09.018 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local 'ceph_args=--fsid=973e3fc1-e63e-4bde-9db6-84c68fd5e93e --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2019-12-21T16:02:09.018 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2019-12-21T16:02:09.018 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-journal-size=100' 2019-12-21T16:02:09.018 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2019-12-21T16:02:09.018 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2019-12-21T16:02:09.019 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2019-12-21T16:02:09.019 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --chdir=' 2019-12-21T16:02:09.019 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+= 2019-12-21T16:02:09.019 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2019-12-21T16:02:09.019 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: get_asok_path 2019-12-21T16:02:09.019 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2019-12-21T16:02:09.019 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2019-12-21T16:02:09.020 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2019-12-21T16:02:09.020 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T16:02:09.020 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.119575 2019-12-21T16:02:09.020 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.119575/$cluster-$name.asok' 2019-12-21T16:02:09.020 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.119575/$cluster-$name.asok' 2019-12-21T16:02:09.020 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --debug-osd=20' 2019-12-21T16:02:09.020 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2019-12-21T16:02:09.021 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2019-12-21T16:02:09.021 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2019-12-21T16:02:09.021 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2019-12-21T16:02:09.021 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2019-12-21T16:02:09.021 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' ' 2019-12-21T16:02:09.022 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2019-12-21T16:02:09.022 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: mkdir -p td/osd-scrub-repair/0 2019-12-21T16:02:09.022 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: echo start osd.0 2019-12-21T16:02:09.022 INFO:tasks.workunit.client.0.smithi172.stderr:start osd.0 2019-12-21T16:02:09.023 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph-osd -i 0 --fsid=973e3fc1-e63e-4bde-9db6-84c68fd5e93e --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.119575/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2019-12-21T16:02:09.023 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: cat td/osd-scrub-repair/0/whoami 2019-12-21T16:02:09.023 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: '[' 0 = 0 ']' 2019-12-21T16:02:09.023 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph osd dump --format=json 2019-12-21T16:02:09.023 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: jq '.flags_set[]' 2019-12-21T16:02:09.024 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: grep -q '"noup"' 2019-12-21T16:02:09.041 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:02:09.039+0000 7f0882903ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T16:02:09.044 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:02:09.041+0000 7f0882903ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T16:02:09.046 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:02:09.044+0000 7f0882903ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T16:02:09.259 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: wait_for_osd up 0 2019-12-21T16:02:09.259 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:914: wait_for_osd: local state=up 2019-12-21T16:02:09.259 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local id=0 2019-12-21T16:02:09.260 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: status=1 2019-12-21T16:02:09.260 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i=0 )) 2019-12-21T16:02:09.260 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i < 300 )) 2019-12-21T16:02:09.260 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: echo 0 2019-12-21T16:02:09.260 INFO:tasks.workunit.client.0.smithi172.stderr:0 2019-12-21T16:02:09.260 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: ceph osd dump 2019-12-21T16:02:09.260 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: grep 'osd.0 up' 2019-12-21T16:02:09.486 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: sleep 1 2019-12-21T16:02:09.562 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:02:09.560+0000 7f0882903ec0 -1 Falling back to public interface 2019-12-21T16:02:10.312 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:02:10.310+0000 7f0882903ec0 -1 osd.0 21 log_to_monitors {default=true} 2019-12-21T16:02:10.488 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i++ )) 2019-12-21T16:02:10.488 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i < 300 )) 2019-12-21T16:02:10.488 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: echo 1 2019-12-21T16:02:10.488 INFO:tasks.workunit.client.0.smithi172.stderr:1 2019-12-21T16:02:10.489 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: ceph osd dump 2019-12-21T16:02:10.489 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: grep 'osd.0 up' 2019-12-21T16:02:10.742 INFO:tasks.workunit.client.0.smithi172.stderr:osd.0 up in weight 1 up_from 24 up_thru 0 down_at 22 last_clean_interval [5,21) [v2:127.0.0.1:6802/142565,v1:127.0.0.1:6803/142565] [v2:127.0.0.1:6804/142565,v1:127.0.0.1:6805/142565] exists,up ab6e82fb-01d4-4001-a78c-831e1be0fa3c 2019-12-21T16:02:10.742 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=0 2019-12-21T16:02:10.742 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: break 2019-12-21T16:02:10.742 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: return 0 2019-12-21T16:02:10.743 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1239: objectstore_tool: wait_for_clean 2019-12-21T16:02:10.743 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1548: wait_for_clean: local cmd= 2019-12-21T16:02:10.743 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1549: wait_for_clean: local num_active_clean=-1 2019-12-21T16:02:10.743 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1550: wait_for_clean: local cur_active_clean 2019-12-21T16:02:10.743 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1551: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2019-12-21T16:02:10.743 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1551: wait_for_clean: get_timeout_delays 90 .1 2019-12-21T16:02:10.743 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1494: get_timeout_delays: shopt -q -o xtrace 2019-12-21T16:02:10.744 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1494: get_timeout_delays: echo true 2019-12-21T16:02:10.744 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1494: get_timeout_delays: local trace=true 2019-12-21T16:02:10.744 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1495: get_timeout_delays: true 2019-12-21T16:02:10.744 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1495: get_timeout_delays: shopt -u -o xtrace 2019-12-21T16:02:10.835 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1551: wait_for_clean: local -a delays 2019-12-21T16:02:10.835 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1552: wait_for_clean: local -i loop=0 2019-12-21T16:02:10.835 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1554: wait_for_clean: flush_pg_stats 2019-12-21T16:02:10.835 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1977: flush_pg_stats: local timeout=300 2019-12-21T16:02:10.835 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1979: flush_pg_stats: ceph osd ls 2019-12-21T16:02:11.061 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1979: flush_pg_stats: ids='0 2019-12-21T16:02:11.061 INFO:tasks.workunit.client.0.smithi172.stderr:1 2019-12-21T16:02:11.061 INFO:tasks.workunit.client.0.smithi172.stderr:2' 2019-12-21T16:02:11.061 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1980: flush_pg_stats: seqs= 2019-12-21T16:02:11.061 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: flush_pg_stats: for osd in $ids 2019-12-21T16:02:11.061 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2019-12-21T16:02:11.171 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: seq=103079215106 2019-12-21T16:02:11.172 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: flush_pg_stats: seqs=' 0-103079215106' 2019-12-21T16:02:11.172 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: flush_pg_stats: for osd in $ids 2019-12-21T16:02:11.172 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2019-12-21T16:02:11.282 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: seq=42949672966 2019-12-21T16:02:11.282 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: flush_pg_stats: seqs=' 0-103079215106 1-42949672966' 2019-12-21T16:02:11.282 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: flush_pg_stats: for osd in $ids 2019-12-21T16:02:11.282 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2019-12-21T16:02:11.392 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: seq=64424509444 2019-12-21T16:02:11.393 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: flush_pg_stats: seqs=' 0-103079215106 1-42949672966 2-64424509444' 2019-12-21T16:02:11.393 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: flush_pg_stats: for s in $seqs 2019-12-21T16:02:11.393 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: echo 0-103079215106 2019-12-21T16:02:11.393 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: cut -d - -f 1 2019-12-21T16:02:11.394 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: osd=0 2019-12-21T16:02:11.394 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: echo 0-103079215106 2019-12-21T16:02:11.394 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: cut -d - -f 2 2019-12-21T16:02:11.395 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: seq=103079215106 2019-12-21T16:02:11.395 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1989: flush_pg_stats: echo 'waiting osd.0 seq 103079215106' 2019-12-21T16:02:11.396 INFO:tasks.workunit.client.0.smithi172.stderr:waiting osd.0 seq 103079215106 2019-12-21T16:02:11.396 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 0 2019-12-21T16:02:11.623 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 21474836485 -lt 103079215106 2019-12-21T16:02:11.623 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1991: flush_pg_stats: sleep 1 2019-12-21T16:02:12.624 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1992: flush_pg_stats: '[' 300 -eq 0 ']' 2019-12-21T16:02:12.625 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 0 2019-12-21T16:02:12.862 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 103079215106 -lt 103079215106 2019-12-21T16:02:12.863 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: flush_pg_stats: for s in $seqs 2019-12-21T16:02:12.863 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: echo 1-42949672966 2019-12-21T16:02:12.863 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: cut -d - -f 1 2019-12-21T16:02:12.864 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: osd=1 2019-12-21T16:02:12.864 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: echo 1-42949672966 2019-12-21T16:02:12.865 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: cut -d - -f 2 2019-12-21T16:02:12.865 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: seq=42949672966 2019-12-21T16:02:12.865 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1989: flush_pg_stats: echo 'waiting osd.1 seq 42949672966' 2019-12-21T16:02:12.866 INFO:tasks.workunit.client.0.smithi172.stderr:waiting osd.1 seq 42949672966 2019-12-21T16:02:12.866 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 1 2019-12-21T16:02:13.091 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 42949672966 -lt 42949672966 2019-12-21T16:02:13.092 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: flush_pg_stats: for s in $seqs 2019-12-21T16:02:13.092 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: echo 2-64424509444 2019-12-21T16:02:13.092 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: cut -d - -f 1 2019-12-21T16:02:13.093 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: osd=2 2019-12-21T16:02:13.093 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: echo 2-64424509444 2019-12-21T16:02:13.094 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: cut -d - -f 2 2019-12-21T16:02:13.094 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: seq=64424509444 2019-12-21T16:02:13.094 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1989: flush_pg_stats: echo 'waiting osd.2 seq 64424509444' 2019-12-21T16:02:13.095 INFO:tasks.workunit.client.0.smithi172.stderr:waiting osd.2 seq 64424509444 2019-12-21T16:02:13.095 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 2 2019-12-21T16:02:13.322 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 64424509444 -lt 64424509444 2019-12-21T16:02:13.323 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1555: wait_for_clean: get_num_pgs 2019-12-21T16:02:13.323 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: ceph --format json status 2019-12-21T16:02:13.323 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: jq .pgmap.num_pgs 2019-12-21T16:02:13.597 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1555: wait_for_clean: test 1 == 0 2019-12-21T16:02:13.598 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1559: wait_for_clean: true 2019-12-21T16:02:13.598 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1563: wait_for_clean: get_num_active_clean 2019-12-21T16:02:13.598 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: get_num_active_clean: local expression 2019-12-21T16:02:13.598 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2019-12-21T16:02:13.598 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("stale") | not)' 2019-12-21T16:02:13.598 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: ceph --format json pg dump pgs 2019-12-21T16:02:13.598 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2019-12-21T16:02:13.821 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1563: wait_for_clean: cur_active_clean=1 2019-12-21T16:02:13.822 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1564: wait_for_clean: get_num_pgs 2019-12-21T16:02:13.822 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: ceph --format json status 2019-12-21T16:02:13.822 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: jq .pgmap.num_pgs 2019-12-21T16:02:14.090 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1564: wait_for_clean: test 1 = 1 2019-12-21T16:02:14.090 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1564: wait_for_clean: break 2019-12-21T16:02:14.090 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: return 0 2019-12-21T16:02:14.091 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:409: TEST_auto_repair_bluestore_scrub: get_pg testpool SOMETHING 2019-12-21T16:02:14.091 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1029: get_pg: local poolname=testpool 2019-12-21T16:02:14.091 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1030: get_pg: local objectname=SOMETHING 2019-12-21T16:02:14.091 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1032: get_pg: ceph --format json osd map testpool SOMETHING 2019-12-21T16:02:14.091 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1032: get_pg: jq -r .pgid 2019-12-21T16:02:14.321 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:409: TEST_auto_repair_bluestore_scrub: local pgid=1.0 2019-12-21T16:02:14.322 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:410: TEST_auto_repair_bluestore_scrub: get_primary testpool SOMETHING 2019-12-21T16:02:14.322 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_primary: local poolname=testpool 2019-12-21T16:02:14.322 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1133: get_primary: local objectname=SOMETHING 2019-12-21T16:02:14.323 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1135: get_primary: ceph --format json osd map testpool SOMETHING 2019-12-21T16:02:14.323 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1136: get_primary: jq .acting_primary 2019-12-21T16:02:14.552 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:410: TEST_auto_repair_bluestore_scrub: local primary=1 2019-12-21T16:02:14.553 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:411: TEST_auto_repair_bluestore_scrub: get_last_scrub_stamp 1.0 2019-12-21T16:02:14.553 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.0 2019-12-21T16:02:14.553 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T16:02:14.554 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T16:02:14.554 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2019-12-21T16:02:14.777 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:411: TEST_auto_repair_bluestore_scrub: local last_scrub_stamp=2019-12-21T16:02:02.466994+0000 2019-12-21T16:02:14.777 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:412: TEST_auto_repair_bluestore_scrub: ceph tell 1.0 scrub 2019-12-21T16:02:14.883 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2019-12-21T16:02:14.883 INFO:tasks.workunit.client.0.smithi172.stdout: "deep": false, 2019-12-21T16:02:14.883 INFO:tasks.workunit.client.0.smithi172.stdout: "stamp": "2019-12-14T16:00:34.882210+0000" 2019-12-21T16:02:14.883 INFO:tasks.workunit.client.0.smithi172.stdout:} 2019-12-21T16:02:14.892 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:415: TEST_auto_repair_bluestore_scrub: wait_for_scrub 1.0 2019-12-21T16:02:02.466994+0000 2019-12-21T16:02:14.892 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1789: wait_for_scrub: local pgid=1.0 2019-12-21T16:02:14.892 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1790: wait_for_scrub: local last_scrub=2019-12-21T16:02:02.466994+0000 2019-12-21T16:02:14.893 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1791: wait_for_scrub: local sname=last_scrub_stamp 2019-12-21T16:02:14.893 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i=0 )) 2019-12-21T16:02:14.893 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T16:02:14.893 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2019-12-21T16:02:14.893 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.0 2019-12-21T16:02:14.893 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T16:02:14.893 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T16:02:14.894 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2019-12-21T16:02:15.113 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T16:02:02.466994+0000 '>' 2019-12-21T16:02:02.466994+0000 2019-12-21T16:02:15.114 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T16:02:16.115 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T16:02:16.115 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T16:02:16.116 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2019-12-21T16:02:16.116 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.0 2019-12-21T16:02:16.116 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T16:02:16.116 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T16:02:16.116 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2019-12-21T16:02:16.347 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-14T16:00:34.882210+0000 '>' 2019-12-21T16:02:02.466994+0000 2019-12-21T16:02:16.347 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T16:02:17.348 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T16:02:17.348 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T16:02:17.349 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2019-12-21T16:02:17.349 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.0 2019-12-21T16:02:17.349 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T16:02:17.350 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T16:02:17.350 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2019-12-21T16:02:17.579 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-14T16:00:34.882210+0000 '>' 2019-12-21T16:02:02.466994+0000 2019-12-21T16:02:17.580 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T16:02:18.580 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T16:02:18.581 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T16:02:18.582 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2019-12-21T16:02:18.582 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.0 2019-12-21T16:02:18.582 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T16:02:18.582 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T16:02:18.582 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2019-12-21T16:02:18.815 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-14T16:00:34.882210+0000 '>' 2019-12-21T16:02:02.466994+0000 2019-12-21T16:02:18.815 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T16:02:19.816 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T16:02:19.816 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T16:02:19.817 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2019-12-21T16:02:19.817 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.0 2019-12-21T16:02:19.817 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T16:02:19.818 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T16:02:19.818 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2019-12-21T16:02:20.041 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-14T16:00:34.882210+0000 '>' 2019-12-21T16:02:02.466994+0000 2019-12-21T16:02:20.042 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T16:02:21.043 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T16:02:21.043 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T16:02:21.044 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2019-12-21T16:02:21.044 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.0 2019-12-21T16:02:21.044 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T16:02:21.044 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T16:02:21.045 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2019-12-21T16:02:21.275 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-14T16:00:34.882210+0000 '>' 2019-12-21T16:02:02.466994+0000 2019-12-21T16:02:21.411 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T16:02:22.277 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T16:02:22.277 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T16:02:22.277 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2019-12-21T16:02:22.277 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=1.0 2019-12-21T16:02:22.278 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T16:02:22.278 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T16:02:22.278 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2019-12-21T16:02:22.516 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T16:02:17.603056+0000 '>' 2019-12-21T16:02:02.466994+0000 2019-12-21T16:02:22.517 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1795: wait_for_scrub: return 0 2019-12-21T16:02:22.517 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:416: TEST_auto_repair_bluestore_scrub: ceph pg dump pgs 2019-12-21T16:02:22.728 INFO:tasks.workunit.client.0.smithi172.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2019-12-21T16:02:22.728 INFO:tasks.workunit.client.0.smithi172.stdout:1.0 1 0 0 0 0 7 0 0 1 1 active+clean 2019-12-21T16:02:17.833921+0000 21'1 25:51 [1,0] 1 [1,0] 1 21'1 2019-12-21T16:02:17.603056+0000 21'1 2019-12-21T16:02:17.603056+0000 0 2019-12-21T16:02:22.728 INFO:tasks.workunit.client.0.smithi172.stdout: 2019-12-21T16:02:22.728 INFO:tasks.workunit.client.0.smithi172.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilisation. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2019-12-21T16:02:22.729 INFO:tasks.workunit.client.0.smithi172.stderr:dumped pgs 2019-12-21T16:02:22.737 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:418: TEST_auto_repair_bluestore_scrub: sleep 5 2019-12-21T16:02:27.739 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:419: TEST_auto_repair_bluestore_scrub: wait_for_clean 2019-12-21T16:02:27.739 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1548: wait_for_clean: local cmd= 2019-12-21T16:02:27.739 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1549: wait_for_clean: local num_active_clean=-1 2019-12-21T16:02:27.739 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1550: wait_for_clean: local cur_active_clean 2019-12-21T16:02:27.739 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1551: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2019-12-21T16:02:27.739 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1551: wait_for_clean: get_timeout_delays 90 .1 2019-12-21T16:02:27.740 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1494: get_timeout_delays: shopt -q -o xtrace 2019-12-21T16:02:27.740 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1494: get_timeout_delays: echo true 2019-12-21T16:02:27.741 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1494: get_timeout_delays: local trace=true 2019-12-21T16:02:27.741 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1495: get_timeout_delays: true 2019-12-21T16:02:27.741 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1495: get_timeout_delays: shopt -u -o xtrace 2019-12-21T16:02:27.845 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1551: wait_for_clean: local -a delays 2019-12-21T16:02:27.845 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1552: wait_for_clean: local -i loop=0 2019-12-21T16:02:27.845 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1554: wait_for_clean: flush_pg_stats 2019-12-21T16:02:27.845 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1977: flush_pg_stats: local timeout=300 2019-12-21T16:02:27.846 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1979: flush_pg_stats: ceph osd ls 2019-12-21T16:02:28.073 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1979: flush_pg_stats: ids='0 2019-12-21T16:02:28.073 INFO:tasks.workunit.client.0.smithi172.stderr:1 2019-12-21T16:02:28.074 INFO:tasks.workunit.client.0.smithi172.stderr:2' 2019-12-21T16:02:28.074 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1980: flush_pg_stats: seqs= 2019-12-21T16:02:28.074 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: flush_pg_stats: for osd in $ids 2019-12-21T16:02:28.074 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2019-12-21T16:02:28.184 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: seq=103079215110 2019-12-21T16:02:28.184 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: flush_pg_stats: seqs=' 0-103079215110' 2019-12-21T16:02:28.185 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: flush_pg_stats: for osd in $ids 2019-12-21T16:02:28.185 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2019-12-21T16:02:28.296 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: seq=42949672970 2019-12-21T16:02:28.296 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: flush_pg_stats: seqs=' 0-103079215110 1-42949672970' 2019-12-21T16:02:28.296 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: flush_pg_stats: for osd in $ids 2019-12-21T16:02:28.296 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2019-12-21T16:02:28.405 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: seq=64424509449 2019-12-21T16:02:28.405 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: flush_pg_stats: seqs=' 0-103079215110 1-42949672970 2-64424509449' 2019-12-21T16:02:28.405 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: flush_pg_stats: for s in $seqs 2019-12-21T16:02:28.406 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: echo 0-103079215110 2019-12-21T16:02:28.406 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: cut -d - -f 1 2019-12-21T16:02:28.406 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: osd=0 2019-12-21T16:02:28.407 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: echo 0-103079215110 2019-12-21T16:02:28.407 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: cut -d - -f 2 2019-12-21T16:02:28.408 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: seq=103079215110 2019-12-21T16:02:28.408 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1989: flush_pg_stats: echo 'waiting osd.0 seq 103079215110' 2019-12-21T16:02:28.408 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 0 2019-12-21T16:02:28.408 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.0 seq 103079215110 2019-12-21T16:02:28.635 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 103079215109 -lt 103079215110 2019-12-21T16:02:28.635 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1991: flush_pg_stats: sleep 1 2019-12-21T16:02:29.636 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1992: flush_pg_stats: '[' 300 -eq 0 ']' 2019-12-21T16:02:29.637 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 0 2019-12-21T16:02:29.877 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 103079215110 -lt 103079215110 2019-12-21T16:02:29.877 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: flush_pg_stats: for s in $seqs 2019-12-21T16:02:29.877 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: echo 1-42949672970 2019-12-21T16:02:29.878 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: cut -d - -f 1 2019-12-21T16:02:29.879 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: osd=1 2019-12-21T16:02:29.879 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: echo 1-42949672970 2019-12-21T16:02:29.879 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: cut -d - -f 2 2019-12-21T16:02:29.880 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: seq=42949672970 2019-12-21T16:02:29.880 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1989: flush_pg_stats: echo 'waiting osd.1 seq 42949672970' 2019-12-21T16:02:29.880 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 1 2019-12-21T16:02:29.880 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.1 seq 42949672970 2019-12-21T16:02:30.105 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 42949672970 -lt 42949672970 2019-12-21T16:02:30.105 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: flush_pg_stats: for s in $seqs 2019-12-21T16:02:30.106 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: echo 2-64424509449 2019-12-21T16:02:30.106 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: cut -d - -f 1 2019-12-21T16:02:30.107 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: osd=2 2019-12-21T16:02:30.107 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: echo 2-64424509449 2019-12-21T16:02:30.108 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: cut -d - -f 2 2019-12-21T16:02:30.109 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: seq=64424509449 2019-12-21T16:02:30.109 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1989: flush_pg_stats: echo 'waiting osd.2 seq 64424509449' 2019-12-21T16:02:30.109 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 2 2019-12-21T16:02:30.109 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.2 seq 64424509449 2019-12-21T16:02:30.336 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 64424509449 -lt 64424509449 2019-12-21T16:02:30.337 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1555: wait_for_clean: get_num_pgs 2019-12-21T16:02:30.337 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: ceph --format json status 2019-12-21T16:02:30.337 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: jq .pgmap.num_pgs 2019-12-21T16:02:30.605 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1555: wait_for_clean: test 1 == 0 2019-12-21T16:02:30.606 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1559: wait_for_clean: true 2019-12-21T16:02:30.606 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1563: wait_for_clean: get_num_active_clean 2019-12-21T16:02:30.606 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: get_num_active_clean: local expression 2019-12-21T16:02:30.606 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2019-12-21T16:02:30.606 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("stale") | not)' 2019-12-21T16:02:30.606 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: ceph --format json pg dump pgs 2019-12-21T16:02:30.607 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2019-12-21T16:02:30.829 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1563: wait_for_clean: cur_active_clean=1 2019-12-21T16:02:30.829 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1564: wait_for_clean: get_num_pgs 2019-12-21T16:02:30.829 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: ceph --format json status 2019-12-21T16:02:30.829 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: jq .pgmap.num_pgs 2019-12-21T16:02:31.097 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1564: wait_for_clean: test 1 = 1 2019-12-21T16:02:31.097 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1564: wait_for_clean: break 2019-12-21T16:02:31.097 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: return 0 2019-12-21T16:02:31.097 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:420: TEST_auto_repair_bluestore_scrub: ceph pg dump pgs 2019-12-21T16:02:31.308 INFO:tasks.workunit.client.0.smithi172.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2019-12-21T16:02:31.309 INFO:tasks.workunit.client.0.smithi172.stdout:1.0 1 0 0 0 0 7 0 0 1 1 active+clean 2019-12-21T16:02:17.833921+0000 21'1 25:51 [1,0] 1 [1,0] 1 21'1 2019-12-21T16:02:17.603056+0000 21'1 2019-12-21T16:02:17.603056+0000 0 2019-12-21T16:02:31.309 INFO:tasks.workunit.client.0.smithi172.stdout: 2019-12-21T16:02:31.309 INFO:tasks.workunit.client.0.smithi172.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilisation. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2019-12-21T16:02:31.310 INFO:tasks.workunit.client.0.smithi172.stderr:dumped pgs 2019-12-21T16:02:31.319 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:423: TEST_auto_repair_bluestore_scrub: get_not_primary testpool SOMETHING 2019-12-21T16:02:31.319 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1165: get_not_primary: local poolname=testpool 2019-12-21T16:02:31.319 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1166: get_not_primary: local objectname=SOMETHING 2019-12-21T16:02:31.320 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: get_not_primary: get_primary testpool SOMETHING 2019-12-21T16:02:31.320 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_primary: local poolname=testpool 2019-12-21T16:02:31.320 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1133: get_primary: local objectname=SOMETHING 2019-12-21T16:02:31.320 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1135: get_primary: ceph --format json osd map testpool SOMETHING 2019-12-21T16:02:31.320 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1136: get_primary: jq .acting_primary 2019-12-21T16:02:31.550 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: get_not_primary: local primary=1 2019-12-21T16:02:31.550 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1169: get_not_primary: ceph --format json osd map testpool SOMETHING 2019-12-21T16:02:31.550 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1170: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2019-12-21T16:02:31.786 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:423: TEST_auto_repair_bluestore_scrub: objectstore_tool td/osd-scrub-repair 0 SOMETHING list-attrs 2019-12-21T16:02:31.786 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: objectstore_tool: local dir=td/osd-scrub-repair 2019-12-21T16:02:31.786 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: objectstore_tool: shift 2019-12-21T16:02:31.786 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1235: objectstore_tool: local id=0 2019-12-21T16:02:31.786 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1236: objectstore_tool: shift 2019-12-21T16:02:31.786 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1238: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING list-attrs 2019-12-21T16:02:31.787 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1205: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2019-12-21T16:02:31.787 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nowait: shift 2019-12-21T16:02:31.787 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1207: _objectstore_tool_nowait: local id=0 2019-12-21T16:02:31.787 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1208: _objectstore_tool_nowait: shift 2019-12-21T16:02:31.787 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1210: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2019-12-21T16:02:31.787 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2019-12-21T16:02:31.787 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2019-12-21T16:02:31.788 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2019-12-21T16:02:31.788 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2019-12-21T16:02:31.788 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2019-12-21T16:02:31.892 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2019-12-21T16:02:31.892 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1212: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING list-attrs 2019-12-21T16:02:31.892 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1193: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2019-12-21T16:02:31.892 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1194: _objectstore_tool_nodown: shift 2019-12-21T16:02:31.893 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1195: _objectstore_tool_nodown: local id=0 2019-12-21T16:02:31.893 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: _objectstore_tool_nodown: shift 2019-12-21T16:02:31.893 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2019-12-21T16:02:31.893 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING list-attrs 2019-12-21T16:02:32.491 INFO:tasks.workunit.client.0.smithi172.stdout:_ 2019-12-21T16:02:32.491 INFO:tasks.workunit.client.0.smithi172.stdout:snapset 2019-12-21T16:02:33.011 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1213: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2019-12-21T16:02:33.011 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:842: activate_osd: local dir=td/osd-scrub-repair 2019-12-21T16:02:33.011 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: shift 2019-12-21T16:02:33.011 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: local id=0 2019-12-21T16:02:33.011 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: shift 2019-12-21T16:02:33.012 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local osd_data=td/osd-scrub-repair/0 2019-12-21T16:02:33.012 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local 'ceph_args=--fsid=973e3fc1-e63e-4bde-9db6-84c68fd5e93e --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2019-12-21T16:02:33.012 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2019-12-21T16:02:33.012 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-journal-size=100' 2019-12-21T16:02:33.012 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2019-12-21T16:02:33.012 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2019-12-21T16:02:33.013 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2019-12-21T16:02:33.013 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --chdir=' 2019-12-21T16:02:33.013 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+= 2019-12-21T16:02:33.013 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2019-12-21T16:02:33.013 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: get_asok_path 2019-12-21T16:02:33.013 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2019-12-21T16:02:33.013 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2019-12-21T16:02:33.014 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2019-12-21T16:02:33.014 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T16:02:33.014 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.119575 2019-12-21T16:02:33.014 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.119575/$cluster-$name.asok' 2019-12-21T16:02:33.014 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.119575/$cluster-$name.asok' 2019-12-21T16:02:33.014 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --debug-osd=20' 2019-12-21T16:02:33.015 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2019-12-21T16:02:33.015 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2019-12-21T16:02:33.015 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2019-12-21T16:02:33.015 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2019-12-21T16:02:33.015 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2019-12-21T16:02:33.015 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' ' 2019-12-21T16:02:33.016 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2019-12-21T16:02:33.016 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: mkdir -p td/osd-scrub-repair/0 2019-12-21T16:02:33.016 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: echo start osd.0 2019-12-21T16:02:33.016 INFO:tasks.workunit.client.0.smithi172.stderr:start osd.0 2019-12-21T16:02:33.019 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph-osd -i 0 --fsid=973e3fc1-e63e-4bde-9db6-84c68fd5e93e --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.119575/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2019-12-21T16:02:33.019 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: cat td/osd-scrub-repair/0/whoami 2019-12-21T16:02:33.021 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: '[' 0 = 0 ']' 2019-12-21T16:02:33.021 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph osd dump --format=json 2019-12-21T16:02:33.022 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: jq '.flags_set[]' 2019-12-21T16:02:33.022 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: grep -q '"noup"' 2019-12-21T16:02:33.032 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:02:33.029+0000 7f5559098ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T16:02:33.033 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:02:33.031+0000 7f5559098ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T16:02:33.035 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:02:33.032+0000 7f5559098ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T16:02:33.242 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: wait_for_osd up 0 2019-12-21T16:02:33.242 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:914: wait_for_osd: local state=up 2019-12-21T16:02:33.243 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local id=0 2019-12-21T16:02:33.243 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: status=1 2019-12-21T16:02:33.243 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i=0 )) 2019-12-21T16:02:33.243 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i < 300 )) 2019-12-21T16:02:33.243 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: echo 0 2019-12-21T16:02:33.243 INFO:tasks.workunit.client.0.smithi172.stderr:0 2019-12-21T16:02:33.243 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: ceph osd dump 2019-12-21T16:02:33.244 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: grep 'osd.0 up' 2019-12-21T16:02:33.470 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: sleep 1 2019-12-21T16:02:33.549 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:02:33.547+0000 7f5559098ec0 -1 Falling back to public interface 2019-12-21T16:02:34.291 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:02:34.288+0000 7f5559098ec0 -1 osd.0 25 log_to_monitors {default=true} 2019-12-21T16:02:34.472 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i++ )) 2019-12-21T16:02:34.472 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i < 300 )) 2019-12-21T16:02:34.472 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: echo 1 2019-12-21T16:02:34.472 INFO:tasks.workunit.client.0.smithi172.stderr:1 2019-12-21T16:02:34.472 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: ceph osd dump 2019-12-21T16:02:34.473 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: grep 'osd.0 up' 2019-12-21T16:02:34.710 INFO:tasks.workunit.client.0.smithi172.stderr:osd.0 up in weight 1 up_from 28 up_thru 0 down_at 26 last_clean_interval [24,25) [v2:127.0.0.1:6802/144036,v1:127.0.0.1:6803/144036] [v2:127.0.0.1:6804/144036,v1:127.0.0.1:6805/144036] exists,up ab6e82fb-01d4-4001-a78c-831e1be0fa3c 2019-12-21T16:02:34.710 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=0 2019-12-21T16:02:34.711 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: break 2019-12-21T16:02:34.711 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: return 0 2019-12-21T16:02:34.711 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1239: objectstore_tool: wait_for_clean 2019-12-21T16:02:34.711 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1548: wait_for_clean: local cmd= 2019-12-21T16:02:34.711 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1549: wait_for_clean: local num_active_clean=-1 2019-12-21T16:02:34.711 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1550: wait_for_clean: local cur_active_clean 2019-12-21T16:02:34.711 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1551: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2019-12-21T16:02:34.712 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1551: wait_for_clean: get_timeout_delays 90 .1 2019-12-21T16:02:34.712 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1494: get_timeout_delays: shopt -q -o xtrace 2019-12-21T16:02:34.712 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1494: get_timeout_delays: echo true 2019-12-21T16:02:34.712 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1494: get_timeout_delays: local trace=true 2019-12-21T16:02:34.712 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1495: get_timeout_delays: true 2019-12-21T16:02:34.712 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1495: get_timeout_delays: shopt -u -o xtrace 2019-12-21T16:02:34.803 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1551: wait_for_clean: local -a delays 2019-12-21T16:02:34.803 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1552: wait_for_clean: local -i loop=0 2019-12-21T16:02:34.804 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1554: wait_for_clean: flush_pg_stats 2019-12-21T16:02:34.804 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1977: flush_pg_stats: local timeout=300 2019-12-21T16:02:34.804 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1979: flush_pg_stats: ceph osd ls 2019-12-21T16:02:35.030 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1979: flush_pg_stats: ids='0 2019-12-21T16:02:35.030 INFO:tasks.workunit.client.0.smithi172.stderr:1 2019-12-21T16:02:35.030 INFO:tasks.workunit.client.0.smithi172.stderr:2' 2019-12-21T16:02:35.031 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1980: flush_pg_stats: seqs= 2019-12-21T16:02:35.031 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: flush_pg_stats: for osd in $ids 2019-12-21T16:02:35.031 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2019-12-21T16:02:35.141 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: seq=120259084290 2019-12-21T16:02:35.142 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: flush_pg_stats: seqs=' 0-120259084290' 2019-12-21T16:02:35.142 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: flush_pg_stats: for osd in $ids 2019-12-21T16:02:35.142 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2019-12-21T16:02:35.252 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: seq=42949672973 2019-12-21T16:02:35.253 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: flush_pg_stats: seqs=' 0-120259084290 1-42949672973' 2019-12-21T16:02:35.253 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: flush_pg_stats: for osd in $ids 2019-12-21T16:02:35.253 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2019-12-21T16:02:35.363 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: seq=64424509451 2019-12-21T16:02:35.364 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: flush_pg_stats: seqs=' 0-120259084290 1-42949672973 2-64424509451' 2019-12-21T16:02:35.364 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: flush_pg_stats: for s in $seqs 2019-12-21T16:02:35.364 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: echo 0-120259084290 2019-12-21T16:02:35.364 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: cut -d - -f 1 2019-12-21T16:02:35.365 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: osd=0 2019-12-21T16:02:35.365 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: echo 0-120259084290 2019-12-21T16:02:35.366 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: cut -d - -f 2 2019-12-21T16:02:35.366 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: seq=120259084290 2019-12-21T16:02:35.366 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1989: flush_pg_stats: echo 'waiting osd.0 seq 120259084290' 2019-12-21T16:02:35.366 INFO:tasks.workunit.client.0.smithi172.stderr:waiting osd.0 seq 120259084290 2019-12-21T16:02:35.367 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 0 2019-12-21T16:02:35.594 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 103079215111 -lt 120259084290 2019-12-21T16:02:35.594 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1991: flush_pg_stats: sleep 1 2019-12-21T16:02:36.595 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1992: flush_pg_stats: '[' 300 -eq 0 ']' 2019-12-21T16:02:36.596 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 0 2019-12-21T16:02:36.834 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 120259084290 -lt 120259084290 2019-12-21T16:02:36.834 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: flush_pg_stats: for s in $seqs 2019-12-21T16:02:36.835 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: echo 1-42949672973 2019-12-21T16:02:36.835 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: cut -d - -f 1 2019-12-21T16:02:36.835 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: osd=1 2019-12-21T16:02:36.836 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: echo 1-42949672973 2019-12-21T16:02:36.836 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: cut -d - -f 2 2019-12-21T16:02:36.837 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: seq=42949672973 2019-12-21T16:02:36.837 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1989: flush_pg_stats: echo 'waiting osd.1 seq 42949672973' 2019-12-21T16:02:36.837 INFO:tasks.workunit.client.0.smithi172.stderr:waiting osd.1 seq 42949672973 2019-12-21T16:02:36.838 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 1 2019-12-21T16:02:37.065 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 42949672973 -lt 42949672973 2019-12-21T16:02:37.065 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: flush_pg_stats: for s in $seqs 2019-12-21T16:02:37.066 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: echo 2-64424509451 2019-12-21T16:02:37.066 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: cut -d - -f 1 2019-12-21T16:02:37.066 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: osd=2 2019-12-21T16:02:37.067 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: echo 2-64424509451 2019-12-21T16:02:37.067 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: cut -d - -f 2 2019-12-21T16:02:37.068 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: seq=64424509451 2019-12-21T16:02:37.068 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1989: flush_pg_stats: echo 'waiting osd.2 seq 64424509451' 2019-12-21T16:02:37.068 INFO:tasks.workunit.client.0.smithi172.stderr:waiting osd.2 seq 64424509451 2019-12-21T16:02:37.068 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 2 2019-12-21T16:02:37.297 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 64424509451 -lt 64424509451 2019-12-21T16:02:37.297 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1555: wait_for_clean: get_num_pgs 2019-12-21T16:02:37.297 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: ceph --format json status 2019-12-21T16:02:37.298 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: jq .pgmap.num_pgs 2019-12-21T16:02:37.565 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1555: wait_for_clean: test 1 == 0 2019-12-21T16:02:37.566 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1559: wait_for_clean: true 2019-12-21T16:02:37.566 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1563: wait_for_clean: get_num_active_clean 2019-12-21T16:02:37.566 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: get_num_active_clean: local expression 2019-12-21T16:02:37.566 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2019-12-21T16:02:37.566 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("stale") | not)' 2019-12-21T16:02:37.566 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: ceph --format json pg dump pgs 2019-12-21T16:02:37.567 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2019-12-21T16:02:37.790 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1563: wait_for_clean: cur_active_clean=1 2019-12-21T16:02:37.790 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1564: wait_for_clean: get_num_pgs 2019-12-21T16:02:37.791 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: ceph --format json status 2019-12-21T16:02:37.791 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: jq .pgmap.num_pgs 2019-12-21T16:02:38.058 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1564: wait_for_clean: test 1 = 1 2019-12-21T16:02:38.058 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1564: wait_for_clean: break 2019-12-21T16:02:38.058 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: return 0 2019-12-21T16:02:38.058 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:424: TEST_auto_repair_bluestore_scrub: rados --pool testpool get SOMETHING td/osd-scrub-repair/COPY 2019-12-21T16:02:38.080 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:425: TEST_auto_repair_bluestore_scrub: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2019-12-21T16:02:38.081 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:426: TEST_auto_repair_bluestore_scrub: grep scrub_finish td/osd-scrub-repair/osd.1.log 2019-12-21T16:02:38.082 INFO:tasks.workunit.client.0.smithi172.stdout:2019-12-21T16:02:16.631+0000 7f91f9877700 20 osd.1 pg_epoch: 25 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=24/25 n=1 ec=17/17 lis/c=24/24 les/c/f=25/25/0 sis=24) [1,0] r=0 lpr=24 crt=21'1 lcod 0'0 mlcod 0'0 active+clean+scrubbing DEEP_SCRUB_ON_ERROR] scrub_finish 2019-12-21T16:02:38.082 INFO:tasks.workunit.client.0.smithi172.stdout:2019-12-21T16:02:16.631+0000 7f91f9877700 20 osd.1 pg_epoch: 25 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=24/25 n=1 ec=17/17 lis/c=24/24 les/c/f=25/25/0 sis=24) [1,0] r=0 lpr=24 crt=21'1 lcod 0'0 mlcod 0'0 active+clean+scrubbing DEEP_SCRUB_ON_ERROR] scrub_finish Try to auto repair after scrub errors 2019-12-21T16:02:38.082 INFO:tasks.workunit.client.0.smithi172.stdout:2019-12-21T16:02:17.602+0000 7f91f9877700 20 osd.1 pg_epoch: 25 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=24/25 n=1 ec=17/17 lis/c=24/24 les/c/f=25/25/0 sis=24) [1,0] r=0 lpr=24 crt=21'1 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent+repair AUTO_REPAIR] scrub_finish 2019-12-21T16:02:38.083 INFO:tasks.workunit.client.0.smithi172.stdout:2019-12-21T16:02:17.602+0000 7f91f9877700 20 osd.1 pg_epoch: 25 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=24/25 n=1 ec=17/17 lis/c=24/24 les/c/f=25/25/0 sis=24) [1,0] r=0 lpr=24 crt=21'1 lcod 0'0 mlcod 0'0 active+scrubbing+deep+inconsistent+repair AUTO_REPAIR mbc={255={(1+0)=1}}] scrub_finish All may be fixed 2019-12-21T16:02:38.083 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:429: TEST_auto_repair_bluestore_scrub: ceph pg 1.0 query 2019-12-21T16:02:38.083 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:429: TEST_auto_repair_bluestore_scrub: jq .info.stats.stat_sum.num_objects_repaired 2019-12-21T16:02:38.198 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:429: TEST_auto_repair_bluestore_scrub: COUNT=1 2019-12-21T16:02:38.199 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:430: TEST_auto_repair_bluestore_scrub: test 1 = 1 2019-12-21T16:02:38.199 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:433: TEST_auto_repair_bluestore_scrub: teardown td/osd-scrub-repair 2019-12-21T16:02:38.199 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2019-12-21T16:02:38.199 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2019-12-21T16:02:38.199 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2019-12-21T16:02:38.199 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2019-12-21T16:02:38.199 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2019-12-21T16:02:38.200 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2019-12-21T16:02:38.200 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2019-12-21T16:02:38.200 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2019-12-21T16:02:38.305 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2019-12-21T16:02:38.306 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2019-12-21T16:02:38.306 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2019-12-21T16:02:38.307 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2019-12-21T16:02:38.307 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2019-12-21T16:02:38.307 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2019-12-21T16:02:38.308 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2019-12-21T16:02:38.309 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local 'pattern=|/usr/lib/systemd/systemd-coredump %P %u %g %s %t %c %h %e' 2019-12-21T16:02:38.310 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' '|' = '|' ']' 2019-12-21T16:02:38.310 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:177: teardown: pattern= 2019-12-21T16:02:38.310 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2019-12-21T16:02:38.310 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname '' 2019-12-21T16:02:38.310 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls . 2019-12-21T16:02:38.311 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2019-12-21T16:02:38.311 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2019-12-21T16:02:38.317 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2019-12-21T16:02:38.317 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T16:02:38.317 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.119575 2019-12-21T16:02:38.317 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.119575 2019-12-21T16:02:38.319 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2019-12-21T16:02:38.320 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2019-12-21T16:02:38.320 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:62: run: for func in $funcs 2019-12-21T16:02:38.320 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: TEST_auto_repair_erasure_coded_appends td/osd-scrub-repair 2019-12-21T16:02:38.320 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:323: TEST_auto_repair_erasure_coded_appends: auto_repair_erasure_coded td/osd-scrub-repair false 2019-12-21T16:02:38.320 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:274: auto_repair_erasure_coded: local dir=td/osd-scrub-repair 2019-12-21T16:02:38.320 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:275: auto_repair_erasure_coded: local allow_overwrites=false 2019-12-21T16:02:38.320 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:276: auto_repair_erasure_coded: local poolname=ecpool 2019-12-21T16:02:38.321 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:279: auto_repair_erasure_coded: setup td/osd-scrub-repair 2019-12-21T16:02:38.321 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2019-12-21T16:02:38.321 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2019-12-21T16:02:38.321 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2019-12-21T16:02:38.321 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2019-12-21T16:02:38.321 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2019-12-21T16:02:38.321 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2019-12-21T16:02:38.322 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2019-12-21T16:02:38.322 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2019-12-21T16:02:38.322 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2019-12-21T16:02:38.322 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2019-12-21T16:02:38.323 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2019-12-21T16:02:38.323 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2019-12-21T16:02:38.323 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2019-12-21T16:02:38.323 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2019-12-21T16:02:38.324 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2019-12-21T16:02:38.324 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2019-12-21T16:02:38.324 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2019-12-21T16:02:38.325 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local 'pattern=|/usr/lib/systemd/systemd-coredump %P %u %g %s %t %c %h %e' 2019-12-21T16:02:38.325 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' '|' = '|' ']' 2019-12-21T16:02:38.325 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:177: teardown: pattern= 2019-12-21T16:02:38.325 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2019-12-21T16:02:38.325 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname '' 2019-12-21T16:02:38.326 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls . 2019-12-21T16:02:38.326 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2019-12-21T16:02:38.326 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2019-12-21T16:02:38.327 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2019-12-21T16:02:38.327 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T16:02:38.327 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.119575 2019-12-21T16:02:38.327 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.119575 2019-12-21T16:02:38.328 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2019-12-21T16:02:38.328 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2019-12-21T16:02:38.328 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2019-12-21T16:02:38.329 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2019-12-21T16:02:38.329 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T16:02:38.329 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.119575 2019-12-21T16:02:38.330 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.119575 2019-12-21T16:02:38.331 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2019-12-21T16:02:38.331 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2019-12-21T16:02:38.332 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2019-12-21T16:02:38.332 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2019-12-21T16:02:38.332 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:280: auto_repair_erasure_coded: run_mon td/osd-scrub-repair a 2019-12-21T16:02:38.332 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2019-12-21T16:02:38.332 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2019-12-21T16:02:38.332 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2019-12-21T16:02:38.332 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2019-12-21T16:02:38.333 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2019-12-21T16:02:38.333 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2019-12-21T16:02:38.600 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2019-12-21T16:02:38.600 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2019-12-21T16:02:38.600 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2019-12-21T16:02:38.600 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2019-12-21T16:02:38.600 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T16:02:38.600 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.119575 2019-12-21T16:02:38.601 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.119575/$cluster-$name.asok' 2019-12-21T16:02:38.601 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.119575/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 2019-12-21T16:02:38.800 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: cat 2019-12-21T16:02:38.801 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: get_config mon a fsid 2019-12-21T16:02:38.801 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: get_config: local daemon=mon 2019-12-21T16:02:38.801 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: get_config: local id=a 2019-12-21T16:02:38.802 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local config=fsid 2019-12-21T16:02:38.802 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: jq -r .fsid 2019-12-21T16:02:38.803 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: get_asok_path mon.a 2019-12-21T16:02:38.803 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2019-12-21T16:02:38.803 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2019-12-21T16:02:38.803 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T16:02:38.804 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T16:02:38.804 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.119575 2019-12-21T16:02:38.804 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.119575/ceph-mon.a.asok 2019-12-21T16:02:38.804 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: CEPH_ARGS= 2019-12-21T16:02:38.804 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: ceph --format json daemon /tmp/ceph-asok.119575/ceph-mon.a.asok config get fsid 2019-12-21T16:02:38.890 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: get_config mon a mon_host 2019-12-21T16:02:38.891 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: get_config: local daemon=mon 2019-12-21T16:02:38.891 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: get_config: local id=a 2019-12-21T16:02:38.891 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local config=mon_host 2019-12-21T16:02:38.891 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: jq -r .mon_host 2019-12-21T16:02:38.891 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: get_asok_path mon.a 2019-12-21T16:02:38.891 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2019-12-21T16:02:38.892 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2019-12-21T16:02:38.892 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T16:02:38.892 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T16:02:38.892 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.119575 2019-12-21T16:02:38.892 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.119575/ceph-mon.a.asok 2019-12-21T16:02:38.893 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: CEPH_ARGS= 2019-12-21T16:02:38.893 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: ceph --format json daemon /tmp/ceph-asok.119575/ceph-mon.a.asok config get mon_host 2019-12-21T16:02:38.979 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:281: auto_repair_erasure_coded: run_mgr td/osd-scrub-repair x 2019-12-21T16:02:38.979 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:551: run_mgr: local dir=td/osd-scrub-repair 2019-12-21T16:02:38.980 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: shift 2019-12-21T16:02:38.980 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local id=x 2019-12-21T16:02:38.980 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2019-12-21T16:02:38.980 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local data=td/osd-scrub-repair/x 2019-12-21T16:02:38.980 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:558: run_mgr: get_asok_path 2019-12-21T16:02:38.980 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2019-12-21T16:02:38.981 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2019-12-21T16:02:38.981 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2019-12-21T16:02:38.981 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T16:02:38.981 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.119575 2019-12-21T16:02:38.981 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.119575/$cluster-$name.asok' 2019-12-21T16:02:38.981 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:558: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2019-12-21T16:02:38.982 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:558: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.119575/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2019-12-21T16:02:39.201 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:286: auto_repair_erasure_coded: local 'ceph_osd_args=--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2019-12-21T16:02:39.201 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:287: auto_repair_erasure_coded: seq 0 2 2019-12-21T16:02:39.205 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:287: auto_repair_erasure_coded: for id in $(seq 0 2) 2019-12-21T16:02:39.205 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:288: auto_repair_erasure_coded: '[' false = true ']' 2019-12-21T16:02:39.205 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:291: auto_repair_erasure_coded: run_osd_filestore td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2019-12-21T16:02:39.205 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:685: run_osd_filestore: local dir=td/osd-scrub-repair 2019-12-21T16:02:39.205 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:686: run_osd_filestore: shift 2019-12-21T16:02:39.206 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:687: run_osd_filestore: local id=0 2019-12-21T16:02:39.206 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:688: run_osd_filestore: shift 2019-12-21T16:02:39.206 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:689: run_osd_filestore: local osd_data=td/osd-scrub-repair/0 2019-12-21T16:02:39.206 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:691: run_osd_filestore: local 'ceph_args=--fsid=973e3fc1-e63e-4bde-9db6-84c68fd5e93e --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2019-12-21T16:02:39.206 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:692: run_osd_filestore: ceph_args+=' --osd-failsafe-full-ratio=.99' 2019-12-21T16:02:39.206 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:693: run_osd_filestore: ceph_args+=' --osd-journal-size=100' 2019-12-21T16:02:39.207 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:694: run_osd_filestore: ceph_args+=' --osd-scrub-load-threshold=2000' 2019-12-21T16:02:39.207 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:695: run_osd_filestore: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2019-12-21T16:02:39.207 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:696: run_osd_filestore: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2019-12-21T16:02:39.207 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:697: run_osd_filestore: ceph_args+=' --chdir=' 2019-12-21T16:02:39.207 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:698: run_osd_filestore: ceph_args+= 2019-12-21T16:02:39.207 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:699: run_osd_filestore: ceph_args+=' --run-dir=td/osd-scrub-repair' 2019-12-21T16:02:39.208 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:700: run_osd_filestore: get_asok_path 2019-12-21T16:02:39.208 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2019-12-21T16:02:39.208 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2019-12-21T16:02:39.208 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2019-12-21T16:02:39.210 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T16:02:39.210 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.119575 2019-12-21T16:02:39.210 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.119575/$cluster-$name.asok' 2019-12-21T16:02:39.211 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:700: run_osd_filestore: ceph_args+=' --admin-socket=/tmp/ceph-asok.119575/$cluster-$name.asok' 2019-12-21T16:02:39.211 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:701: run_osd_filestore: ceph_args+=' --debug-osd=20' 2019-12-21T16:02:39.211 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:702: run_osd_filestore: ceph_args+=' --debug-ms=1' 2019-12-21T16:02:39.211 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:703: run_osd_filestore: ceph_args+=' --debug-monc=20' 2019-12-21T16:02:39.211 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:704: run_osd_filestore: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2019-12-21T16:02:39.211 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:705: run_osd_filestore: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2019-12-21T16:02:39.212 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:706: run_osd_filestore: ceph_args+=' --osd-max-object-name-len=460' 2019-12-21T16:02:39.212 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:707: run_osd_filestore: ceph_args+=' --osd-max-object-namespace-len=64' 2019-12-21T16:02:39.212 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:708: run_osd_filestore: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2019-12-21T16:02:39.212 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:709: run_osd_filestore: ceph_args+=' ' 2019-12-21T16:02:39.214 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:710: run_osd_filestore: ceph_args+='--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2019-12-21T16:02:39.214 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:711: run_osd_filestore: mkdir -p td/osd-scrub-repair/0 2019-12-21T16:02:39.214 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:713: run_osd_filestore: uuidgen 2019-12-21T16:02:39.215 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:713: run_osd_filestore: local uuid=077d7ea3-8757-489e-b000-154f6861f322 2019-12-21T16:02:39.215 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:714: run_osd_filestore: echo 'add osd2 077d7ea3-8757-489e-b000-154f6861f322' 2019-12-21T16:02:39.215 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:715: run_osd_filestore: ceph-authtool --gen-print-key 2019-12-21T16:02:39.215 INFO:tasks.workunit.client.0.smithi172.stdout:add osd2 077d7ea3-8757-489e-b000-154f6861f322 2019-12-21T16:02:39.226 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:715: run_osd_filestore: OSD_SECRET=AQAfQv5dWAJfDRAAb5/eCq6+6MCv38yVKCItDw== 2019-12-21T16:02:39.226 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:716: run_osd_filestore: echo '{"cephx_secret": "AQAfQv5dWAJfDRAAb5/eCq6+6MCv38yVKCItDw=="}' 2019-12-21T16:02:39.227 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:717: run_osd_filestore: ceph osd new 077d7ea3-8757-489e-b000-154f6861f322 -i td/osd-scrub-repair/0/new.json 2019-12-21T16:02:39.483 INFO:tasks.workunit.client.0.smithi172.stdout:0 2019-12-21T16:02:39.493 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:718: run_osd_filestore: rm td/osd-scrub-repair/0/new.json 2019-12-21T16:02:39.494 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:719: run_osd_filestore: ceph-osd -i 0 --fsid=973e3fc1-e63e-4bde-9db6-84c68fd5e93e --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.119575/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQAfQv5dWAJfDRAAb5/eCq6+6MCv38yVKCItDw== --osd-uuid 077d7ea3-8757-489e-b000-154f6861f322 --osd-objectstore=filestore 2019-12-21T16:02:39.519 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:02:39.516+0000 7fd6d80d0ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T16:02:39.520 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:02:39.518+0000 7fd6d80d0ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T16:02:39.521 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:02:39.519+0000 7fd6d80d0ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T16:02:39.818 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:02:39.816+0000 7fd6d80d0ec0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2019-12-21T16:02:40.188 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:02:40.185+0000 7fd6d80d0ec0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2019-12-21T16:02:40.188 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:02:40.185+0000 7fd6d80d0ec0 -1 journal do_read_entry(4096): bad header magic 2019-12-21T16:02:40.188 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:02:40.185+0000 7fd6d80d0ec0 -1 journal do_read_entry(4096): bad header magic 2019-12-21T16:02:40.599 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:721: run_osd_filestore: local key_fn=td/osd-scrub-repair/0/keyring 2019-12-21T16:02:40.599 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:722: run_osd_filestore: cat 2019-12-21T16:02:40.600 INFO:tasks.workunit.client.0.smithi172.stdout:adding osd0 key to auth repository 2019-12-21T16:02:40.600 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:726: run_osd_filestore: echo adding osd0 key to auth repository 2019-12-21T16:02:40.600 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:727: run_osd_filestore: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2019-12-21T16:02:40.773 INFO:tasks.workunit.client.0.smithi172.stderr:Error EINVAL: key for osd.0 not found in provided keyring 2019-12-21T16:02:40.781 INFO:tasks.workunit.client.0.smithi172.stdout:start osd.0 2019-12-21T16:02:40.781 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:728: run_osd_filestore: echo start osd.0 2019-12-21T16:02:40.781 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:732: run_osd_filestore: ceph osd dump --format=json 2019-12-21T16:02:40.782 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:729: run_osd_filestore: ceph-osd -i 0 --fsid=973e3fc1-e63e-4bde-9db6-84c68fd5e93e --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.119575/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2019-12-21T16:02:40.782 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:732: run_osd_filestore: jq '.flags_set[]' 2019-12-21T16:02:40.782 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:732: run_osd_filestore: grep -q '"noup"' 2019-12-21T16:02:40.803 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:02:40.801+0000 7f20b9a2fec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T16:02:40.805 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:02:40.803+0000 7f20b9a2fec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T16:02:40.807 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:02:40.805+0000 7f20b9a2fec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T16:02:40.809 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:02:40.806+0000 7f20b9a2fec0 -1 Falling back to public interface 2019-12-21T16:02:40.987 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:735: run_osd_filestore: wait_for_osd up 0 2019-12-21T16:02:40.987 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:914: wait_for_osd: local state=up 2019-12-21T16:02:40.987 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local id=0 2019-12-21T16:02:40.987 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: status=1 2019-12-21T16:02:40.987 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i=0 )) 2019-12-21T16:02:40.987 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i < 300 )) 2019-12-21T16:02:40.988 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: echo 0 2019-12-21T16:02:40.988 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: ceph osd dump 2019-12-21T16:02:40.988 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: grep 'osd.0 up' 2019-12-21T16:02:40.988 INFO:tasks.workunit.client.0.smithi172.stdout:0 2019-12-21T16:02:41.076 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:02:41.073+0000 7f20b9a2fec0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2019-12-21T16:02:41.076 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:02:41.073+0000 7f20b9a2fec0 -1 journal do_read_entry(8192): bad header magic 2019-12-21T16:02:41.076 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:02:41.073+0000 7f20b9a2fec0 -1 journal do_read_entry(8192): bad header magic 2019-12-21T16:02:41.081 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:02:41.078+0000 7f20b9a2fec0 -1 osd.0 0 log_to_monitors {default=true} 2019-12-21T16:02:41.165 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: sleep 1 2019-12-21T16:02:42.163 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i++ )) 2019-12-21T16:02:42.163 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i < 300 )) 2019-12-21T16:02:42.163 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: echo 1 2019-12-21T16:02:42.163 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: ceph osd dump 2019-12-21T16:02:42.163 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: grep 'osd.0 up' 2019-12-21T16:02:42.164 INFO:tasks.workunit.client.0.smithi172.stdout:1 2019-12-21T16:02:42.340 INFO:tasks.workunit.client.0.smithi172.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/144992,v1:127.0.0.1:6801/144992] [v2:127.0.0.1:6802/144992,v1:127.0.0.1:6803/144992] exists,up 077d7ea3-8757-489e-b000-154f6861f322 2019-12-21T16:02:42.340 INFO:tasks.workunit.client.0.smithi172.stdout:add osd2 46c8d16e-77f0-4445-965e-c865f6694611 2019-12-21T16:02:42.340 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=0 2019-12-21T16:02:42.341 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: break 2019-12-21T16:02:42.341 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: return 0 2019-12-21T16:02:42.341 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:287: auto_repair_erasure_coded: for id in $(seq 0 2) 2019-12-21T16:02:42.341 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:288: auto_repair_erasure_coded: '[' false = true ']' 2019-12-21T16:02:42.341 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:291: auto_repair_erasure_coded: run_osd_filestore td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2019-12-21T16:02:42.341 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:685: run_osd_filestore: local dir=td/osd-scrub-repair 2019-12-21T16:02:42.342 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:686: run_osd_filestore: shift 2019-12-21T16:02:42.342 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:687: run_osd_filestore: local id=1 2019-12-21T16:02:42.342 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:688: run_osd_filestore: shift 2019-12-21T16:02:42.342 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:689: run_osd_filestore: local osd_data=td/osd-scrub-repair/1 2019-12-21T16:02:42.342 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:691: run_osd_filestore: local 'ceph_args=--fsid=973e3fc1-e63e-4bde-9db6-84c68fd5e93e --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2019-12-21T16:02:42.342 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:692: run_osd_filestore: ceph_args+=' --osd-failsafe-full-ratio=.99' 2019-12-21T16:02:42.342 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:693: run_osd_filestore: ceph_args+=' --osd-journal-size=100' 2019-12-21T16:02:42.342 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:694: run_osd_filestore: ceph_args+=' --osd-scrub-load-threshold=2000' 2019-12-21T16:02:42.343 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:695: run_osd_filestore: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2019-12-21T16:02:42.343 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:696: run_osd_filestore: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2019-12-21T16:02:42.343 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:697: run_osd_filestore: ceph_args+=' --chdir=' 2019-12-21T16:02:42.343 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:698: run_osd_filestore: ceph_args+= 2019-12-21T16:02:42.343 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:699: run_osd_filestore: ceph_args+=' --run-dir=td/osd-scrub-repair' 2019-12-21T16:02:42.343 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:700: run_osd_filestore: get_asok_path 2019-12-21T16:02:42.343 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2019-12-21T16:02:42.344 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2019-12-21T16:02:42.344 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2019-12-21T16:02:42.344 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T16:02:42.344 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.119575 2019-12-21T16:02:42.344 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.119575/$cluster-$name.asok' 2019-12-21T16:02:42.344 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:700: run_osd_filestore: ceph_args+=' --admin-socket=/tmp/ceph-asok.119575/$cluster-$name.asok' 2019-12-21T16:02:42.344 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:701: run_osd_filestore: ceph_args+=' --debug-osd=20' 2019-12-21T16:02:42.345 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:702: run_osd_filestore: ceph_args+=' --debug-ms=1' 2019-12-21T16:02:42.345 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:703: run_osd_filestore: ceph_args+=' --debug-monc=20' 2019-12-21T16:02:42.345 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:704: run_osd_filestore: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2019-12-21T16:02:42.345 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:705: run_osd_filestore: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2019-12-21T16:02:42.345 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:706: run_osd_filestore: ceph_args+=' --osd-max-object-name-len=460' 2019-12-21T16:02:42.345 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:707: run_osd_filestore: ceph_args+=' --osd-max-object-namespace-len=64' 2019-12-21T16:02:42.345 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:708: run_osd_filestore: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2019-12-21T16:02:42.346 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:709: run_osd_filestore: ceph_args+=' ' 2019-12-21T16:02:42.346 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:710: run_osd_filestore: ceph_args+='--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2019-12-21T16:02:42.346 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:711: run_osd_filestore: mkdir -p td/osd-scrub-repair/1 2019-12-21T16:02:42.346 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:713: run_osd_filestore: uuidgen 2019-12-21T16:02:42.346 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:713: run_osd_filestore: local uuid=46c8d16e-77f0-4445-965e-c865f6694611 2019-12-21T16:02:42.347 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:714: run_osd_filestore: echo 'add osd2 46c8d16e-77f0-4445-965e-c865f6694611' 2019-12-21T16:02:42.347 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:715: run_osd_filestore: ceph-authtool --gen-print-key 2019-12-21T16:02:42.352 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:715: run_osd_filestore: OSD_SECRET=AQAiQv5dmqrnFBAAAIUq0avbnSKBqp6CqlL/SQ== 2019-12-21T16:02:42.353 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:716: run_osd_filestore: echo '{"cephx_secret": "AQAiQv5dmqrnFBAAAIUq0avbnSKBqp6CqlL/SQ=="}' 2019-12-21T16:02:42.353 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:717: run_osd_filestore: ceph osd new 46c8d16e-77f0-4445-965e-c865f6694611 -i td/osd-scrub-repair/1/new.json 2019-12-21T16:02:42.622 INFO:tasks.workunit.client.0.smithi172.stdout:1 2019-12-21T16:02:42.631 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:718: run_osd_filestore: rm td/osd-scrub-repair/1/new.json 2019-12-21T16:02:42.632 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:719: run_osd_filestore: ceph-osd -i 1 --fsid=973e3fc1-e63e-4bde-9db6-84c68fd5e93e --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.119575/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQAiQv5dmqrnFBAAAIUq0avbnSKBqp6CqlL/SQ== --osd-uuid 46c8d16e-77f0-4445-965e-c865f6694611 --osd-objectstore=filestore 2019-12-21T16:02:42.663 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:02:42.654+0000 7f33ba069ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T16:02:42.664 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:02:42.661+0000 7f33ba069ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T16:02:42.667 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:02:42.663+0000 7f33ba069ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T16:02:43.141 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:02:43.138+0000 7f33ba069ec0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2019-12-21T16:02:43.535 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:02:43.533+0000 7f33ba069ec0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2019-12-21T16:02:43.535 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:02:43.533+0000 7f33ba069ec0 -1 journal do_read_entry(4096): bad header magic 2019-12-21T16:02:43.536 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:02:43.533+0000 7f33ba069ec0 -1 journal do_read_entry(4096): bad header magic 2019-12-21T16:02:44.047 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:721: run_osd_filestore: local key_fn=td/osd-scrub-repair/1/keyring 2019-12-21T16:02:44.048 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:722: run_osd_filestore: cat 2019-12-21T16:02:44.049 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:726: run_osd_filestore: echo adding osd1 key to auth repository 2019-12-21T16:02:44.049 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:727: run_osd_filestore: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2019-12-21T16:02:44.049 INFO:tasks.workunit.client.0.smithi172.stdout:adding osd1 key to auth repository 2019-12-21T16:02:44.325 INFO:tasks.workunit.client.0.smithi172.stderr:Error EINVAL: key for osd.1 not found in provided keyring 2019-12-21T16:02:44.333 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:728: run_osd_filestore: echo start osd.1 2019-12-21T16:02:44.334 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:729: run_osd_filestore: ceph-osd -i 1 --fsid=973e3fc1-e63e-4bde-9db6-84c68fd5e93e --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.119575/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2019-12-21T16:02:44.334 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:732: run_osd_filestore: ceph osd dump --format=json 2019-12-21T16:02:44.334 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:732: run_osd_filestore: jq '.flags_set[]' 2019-12-21T16:02:44.334 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:732: run_osd_filestore: grep -q '"noup"' 2019-12-21T16:02:44.334 INFO:tasks.workunit.client.0.smithi172.stdout:start osd.1 2019-12-21T16:02:44.355 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:02:44.352+0000 7f92f93a3ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T16:02:44.357 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:02:44.354+0000 7f92f93a3ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T16:02:44.359 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:02:44.356+0000 7f92f93a3ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T16:02:44.360 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:02:44.358+0000 7f92f93a3ec0 -1 Falling back to public interface 2019-12-21T16:02:44.623 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:02:44.620+0000 7f92f93a3ec0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2019-12-21T16:02:44.623 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:02:44.621+0000 7f92f93a3ec0 -1 journal do_read_entry(8192): bad header magic 2019-12-21T16:02:44.624 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:02:44.621+0000 7f92f93a3ec0 -1 journal do_read_entry(8192): bad header magic 2019-12-21T16:02:44.628 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:02:44.625+0000 7f92f93a3ec0 -1 osd.1 0 log_to_monitors {default=true} 2019-12-21T16:02:44.657 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:735: run_osd_filestore: wait_for_osd up 1 2019-12-21T16:02:44.657 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:914: wait_for_osd: local state=up 2019-12-21T16:02:44.657 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local id=1 2019-12-21T16:02:44.657 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: status=1 2019-12-21T16:02:44.657 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i=0 )) 2019-12-21T16:02:44.658 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i < 300 )) 2019-12-21T16:02:44.658 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: echo 0 2019-12-21T16:02:44.658 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: ceph osd dump 2019-12-21T16:02:44.658 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: grep 'osd.1 up' 2019-12-21T16:02:44.658 INFO:tasks.workunit.client.0.smithi172.stdout:0 2019-12-21T16:02:44.945 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: sleep 1 2019-12-21T16:02:45.947 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i++ )) 2019-12-21T16:02:45.947 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i < 300 )) 2019-12-21T16:02:45.947 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: echo 1 2019-12-21T16:02:45.947 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: ceph osd dump 2019-12-21T16:02:45.947 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: grep 'osd.1 up' 2019-12-21T16:02:45.948 INFO:tasks.workunit.client.0.smithi172.stdout:1 2019-12-21T16:02:46.183 INFO:tasks.workunit.client.0.smithi172.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/145330,v1:127.0.0.1:6811/145330] [v2:127.0.0.1:6812/145330,v1:127.0.0.1:6813/145330] exists,up 46c8d16e-77f0-4445-965e-c865f6694611 2019-12-21T16:02:46.184 INFO:tasks.workunit.client.0.smithi172.stdout:add osd2 e07a8ac5-cacb-4f54-9143-7904cffa475a 2019-12-21T16:02:46.184 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=0 2019-12-21T16:02:46.184 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: break 2019-12-21T16:02:46.184 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: return 0 2019-12-21T16:02:46.185 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:287: auto_repair_erasure_coded: for id in $(seq 0 2) 2019-12-21T16:02:46.185 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:288: auto_repair_erasure_coded: '[' false = true ']' 2019-12-21T16:02:46.185 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:291: auto_repair_erasure_coded: run_osd_filestore td/osd-scrub-repair 2 --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2019-12-21T16:02:46.185 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:685: run_osd_filestore: local dir=td/osd-scrub-repair 2019-12-21T16:02:46.185 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:686: run_osd_filestore: shift 2019-12-21T16:02:46.186 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:687: run_osd_filestore: local id=2 2019-12-21T16:02:46.186 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:688: run_osd_filestore: shift 2019-12-21T16:02:46.186 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:689: run_osd_filestore: local osd_data=td/osd-scrub-repair/2 2019-12-21T16:02:46.186 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:691: run_osd_filestore: local 'ceph_args=--fsid=973e3fc1-e63e-4bde-9db6-84c68fd5e93e --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2019-12-21T16:02:46.186 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:692: run_osd_filestore: ceph_args+=' --osd-failsafe-full-ratio=.99' 2019-12-21T16:02:46.186 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:693: run_osd_filestore: ceph_args+=' --osd-journal-size=100' 2019-12-21T16:02:46.187 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:694: run_osd_filestore: ceph_args+=' --osd-scrub-load-threshold=2000' 2019-12-21T16:02:46.187 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:695: run_osd_filestore: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2019-12-21T16:02:46.187 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:696: run_osd_filestore: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2019-12-21T16:02:46.187 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:697: run_osd_filestore: ceph_args+=' --chdir=' 2019-12-21T16:02:46.187 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:698: run_osd_filestore: ceph_args+= 2019-12-21T16:02:46.187 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:699: run_osd_filestore: ceph_args+=' --run-dir=td/osd-scrub-repair' 2019-12-21T16:02:46.187 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:700: run_osd_filestore: get_asok_path 2019-12-21T16:02:46.188 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2019-12-21T16:02:46.188 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2019-12-21T16:02:46.188 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2019-12-21T16:02:46.188 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T16:02:46.188 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.119575 2019-12-21T16:02:46.188 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.119575/$cluster-$name.asok' 2019-12-21T16:02:46.189 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:700: run_osd_filestore: ceph_args+=' --admin-socket=/tmp/ceph-asok.119575/$cluster-$name.asok' 2019-12-21T16:02:46.189 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:701: run_osd_filestore: ceph_args+=' --debug-osd=20' 2019-12-21T16:02:46.189 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:702: run_osd_filestore: ceph_args+=' --debug-ms=1' 2019-12-21T16:02:46.189 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:703: run_osd_filestore: ceph_args+=' --debug-monc=20' 2019-12-21T16:02:46.189 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:704: run_osd_filestore: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2019-12-21T16:02:46.189 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:705: run_osd_filestore: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2019-12-21T16:02:46.189 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:706: run_osd_filestore: ceph_args+=' --osd-max-object-name-len=460' 2019-12-21T16:02:46.190 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:707: run_osd_filestore: ceph_args+=' --osd-max-object-namespace-len=64' 2019-12-21T16:02:46.190 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:708: run_osd_filestore: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2019-12-21T16:02:46.190 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:709: run_osd_filestore: ceph_args+=' ' 2019-12-21T16:02:46.190 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:710: run_osd_filestore: ceph_args+='--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2019-12-21T16:02:46.190 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:711: run_osd_filestore: mkdir -p td/osd-scrub-repair/2 2019-12-21T16:02:46.190 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:713: run_osd_filestore: uuidgen 2019-12-21T16:02:46.190 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:713: run_osd_filestore: local uuid=e07a8ac5-cacb-4f54-9143-7904cffa475a 2019-12-21T16:02:46.191 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:714: run_osd_filestore: echo 'add osd2 e07a8ac5-cacb-4f54-9143-7904cffa475a' 2019-12-21T16:02:46.191 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:715: run_osd_filestore: ceph-authtool --gen-print-key 2019-12-21T16:02:46.195 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:715: run_osd_filestore: OSD_SECRET=AQAmQv5dHY+LCxAA25zdipM4TL01DYLdweMdkw== 2019-12-21T16:02:46.195 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:716: run_osd_filestore: echo '{"cephx_secret": "AQAmQv5dHY+LCxAA25zdipM4TL01DYLdweMdkw=="}' 2019-12-21T16:02:46.196 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:717: run_osd_filestore: ceph osd new e07a8ac5-cacb-4f54-9143-7904cffa475a -i td/osd-scrub-repair/2/new.json 2019-12-21T16:02:46.521 INFO:tasks.workunit.client.0.smithi172.stdout:2 2019-12-21T16:02:46.531 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:718: run_osd_filestore: rm td/osd-scrub-repair/2/new.json 2019-12-21T16:02:46.532 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:719: run_osd_filestore: ceph-osd -i 2 --fsid=973e3fc1-e63e-4bde-9db6-84c68fd5e93e --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.119575/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQAmQv5dHY+LCxAA25zdipM4TL01DYLdweMdkw== --osd-uuid e07a8ac5-cacb-4f54-9143-7904cffa475a --osd-objectstore=filestore 2019-12-21T16:02:46.556 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:02:46.554+0000 7f2cb2adcec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T16:02:46.562 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:02:46.561+0000 7f2cb2adcec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T16:02:46.564 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:02:46.562+0000 7f2cb2adcec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T16:02:46.973 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:02:46.972+0000 7f2cb2adcec0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2019-12-21T16:02:47.368 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:02:47.366+0000 7f2cb2adcec0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2019-12-21T16:02:47.368 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:02:47.366+0000 7f2cb2adcec0 -1 journal do_read_entry(4096): bad header magic 2019-12-21T16:02:47.368 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:02:47.366+0000 7f2cb2adcec0 -1 journal do_read_entry(4096): bad header magic 2019-12-21T16:02:47.763 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:721: run_osd_filestore: local key_fn=td/osd-scrub-repair/2/keyring 2019-12-21T16:02:47.764 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:722: run_osd_filestore: cat 2019-12-21T16:02:47.766 INFO:tasks.workunit.client.0.smithi172.stdout:adding osd2 key to auth repository 2019-12-21T16:02:47.767 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:726: run_osd_filestore: echo adding osd2 key to auth repository 2019-12-21T16:02:47.767 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:727: run_osd_filestore: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2019-12-21T16:02:48.048 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:728: run_osd_filestore: echo start osd.2 2019-12-21T16:02:48.048 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:729: run_osd_filestore: ceph-osd -i 2 --fsid=973e3fc1-e63e-4bde-9db6-84c68fd5e93e --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.119575/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2019-12-21T16:02:48.048 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:732: run_osd_filestore: ceph osd dump --format=json 2019-12-21T16:02:48.048 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:732: run_osd_filestore: jq '.flags_set[]' 2019-12-21T16:02:48.049 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:732: run_osd_filestore: grep -q '"noup"' 2019-12-21T16:02:48.049 INFO:tasks.workunit.client.0.smithi172.stdout:start osd.2 2019-12-21T16:02:48.070 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:02:48.068+0000 7f88b8c6eec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T16:02:48.087 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:02:48.086+0000 7f88b8c6eec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T16:02:48.089 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:02:48.088+0000 7f88b8c6eec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T16:02:48.091 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:02:48.089+0000 7f88b8c6eec0 -1 Falling back to public interface 2019-12-21T16:02:48.316 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:735: run_osd_filestore: wait_for_osd up 2 2019-12-21T16:02:48.316 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:914: wait_for_osd: local state=up 2019-12-21T16:02:48.317 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local id=2 2019-12-21T16:02:48.317 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: status=1 2019-12-21T16:02:48.317 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i=0 )) 2019-12-21T16:02:48.317 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i < 300 )) 2019-12-21T16:02:48.317 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: echo 0 2019-12-21T16:02:48.317 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: ceph osd dump 2019-12-21T16:02:48.317 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: grep 'osd.2 up' 2019-12-21T16:02:48.318 INFO:tasks.workunit.client.0.smithi172.stdout:0 2019-12-21T16:02:48.398 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:02:48.396+0000 7f88b8c6eec0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2019-12-21T16:02:48.399 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:02:48.396+0000 7f88b8c6eec0 -1 journal do_read_entry(8192): bad header magic 2019-12-21T16:02:48.399 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:02:48.396+0000 7f88b8c6eec0 -1 journal do_read_entry(8192): bad header magic 2019-12-21T16:02:48.403 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:02:48.401+0000 7f88b8c6eec0 -1 osd.2 0 log_to_monitors {default=true} 2019-12-21T16:02:48.601 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: sleep 1 2019-12-21T16:02:49.602 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i++ )) 2019-12-21T16:02:49.603 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i < 300 )) 2019-12-21T16:02:49.603 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: echo 1 2019-12-21T16:02:49.603 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: ceph osd dump 2019-12-21T16:02:49.603 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: grep 'osd.2 up' 2019-12-21T16:02:49.603 INFO:tasks.workunit.client.0.smithi172.stdout:1 2019-12-21T16:02:49.834 INFO:tasks.workunit.client.0.smithi172.stdout:osd.2 up in weight 1 up_from 16 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/145629,v1:127.0.0.1:6819/145629] [v2:127.0.0.1:6820/145629,v1:127.0.0.1:6821/145629] exists,up e07a8ac5-cacb-4f54-9143-7904cffa475a 2019-12-21T16:02:49.835 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=0 2019-12-21T16:02:49.835 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: break 2019-12-21T16:02:49.835 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: return 0 2019-12-21T16:02:49.835 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:294: auto_repair_erasure_coded: create_rbd_pool 2019-12-21T16:02:49.835 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2019-12-21T16:02:50.058 INFO:tasks.workunit.client.0.smithi172.stderr:pool 'rbd' does not exist 2019-12-21T16:02:50.066 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:534: create_rbd_pool: create_pool rbd 4 2019-12-21T16:02:50.066 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:539: create_pool: ceph osd pool create rbd 4 2019-12-21T16:02:50.432 INFO:tasks.workunit.client.0.smithi172.stderr:pool 'rbd' created 2019-12-21T16:02:50.440 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: sleep 1 2019-12-21T16:02:51.442 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: rbd pool init rbd 2019-12-21T16:02:53.045 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:295: auto_repair_erasure_coded: wait_for_clean 2019-12-21T16:02:53.045 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1548: wait_for_clean: local cmd= 2019-12-21T16:02:53.046 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1549: wait_for_clean: local num_active_clean=-1 2019-12-21T16:02:53.046 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1550: wait_for_clean: local cur_active_clean 2019-12-21T16:02:53.046 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1551: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2019-12-21T16:02:53.046 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1551: wait_for_clean: get_timeout_delays 90 .1 2019-12-21T16:02:53.047 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1494: get_timeout_delays: shopt -q -o xtrace 2019-12-21T16:02:53.047 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1494: get_timeout_delays: echo true 2019-12-21T16:02:53.047 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1494: get_timeout_delays: local trace=true 2019-12-21T16:02:53.047 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1495: get_timeout_delays: true 2019-12-21T16:02:53.047 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1495: get_timeout_delays: shopt -u -o xtrace 2019-12-21T16:02:53.152 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1551: wait_for_clean: local -a delays 2019-12-21T16:02:53.152 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1552: wait_for_clean: local -i loop=0 2019-12-21T16:02:53.152 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1554: wait_for_clean: flush_pg_stats 2019-12-21T16:02:53.152 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1977: flush_pg_stats: local timeout=300 2019-12-21T16:02:53.152 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1979: flush_pg_stats: ceph osd ls 2019-12-21T16:02:53.379 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1979: flush_pg_stats: ids='0 2019-12-21T16:02:53.379 INFO:tasks.workunit.client.0.smithi172.stderr:1 2019-12-21T16:02:53.379 INFO:tasks.workunit.client.0.smithi172.stderr:2' 2019-12-21T16:02:53.379 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1980: flush_pg_stats: seqs= 2019-12-21T16:02:53.380 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: flush_pg_stats: for osd in $ids 2019-12-21T16:02:53.380 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2019-12-21T16:02:53.613 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: seq=21474836484 2019-12-21T16:02:53.613 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: flush_pg_stats: seqs=' 0-21474836484' 2019-12-21T16:02:53.613 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: flush_pg_stats: for osd in $ids 2019-12-21T16:02:53.614 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2019-12-21T16:02:53.722 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: seq=42949672964 2019-12-21T16:02:53.722 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: flush_pg_stats: seqs=' 0-21474836484 1-42949672964' 2019-12-21T16:02:53.723 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: flush_pg_stats: for osd in $ids 2019-12-21T16:02:53.723 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2019-12-21T16:02:53.831 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: seq=68719476738 2019-12-21T16:02:53.831 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: flush_pg_stats: seqs=' 0-21474836484 1-42949672964 2-68719476738' 2019-12-21T16:02:53.831 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: flush_pg_stats: for s in $seqs 2019-12-21T16:02:53.832 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: echo 0-21474836484 2019-12-21T16:02:53.832 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: cut -d - -f 1 2019-12-21T16:02:53.833 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: osd=0 2019-12-21T16:02:53.833 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: echo 0-21474836484 2019-12-21T16:02:53.833 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: cut -d - -f 2 2019-12-21T16:02:53.834 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: seq=21474836484 2019-12-21T16:02:53.834 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1989: flush_pg_stats: echo 'waiting osd.0 seq 21474836484' 2019-12-21T16:02:53.834 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 0 2019-12-21T16:02:53.834 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.0 seq 21474836484 2019-12-21T16:02:54.064 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 21474836482 -lt 21474836484 2019-12-21T16:02:54.064 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1991: flush_pg_stats: sleep 1 2019-12-21T16:02:55.065 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1992: flush_pg_stats: '[' 300 -eq 0 ']' 2019-12-21T16:02:55.066 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 0 2019-12-21T16:02:55.293 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 21474836484 -lt 21474836484 2019-12-21T16:02:55.294 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: flush_pg_stats: for s in $seqs 2019-12-21T16:02:55.294 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: echo 1-42949672964 2019-12-21T16:02:55.294 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: cut -d - -f 1 2019-12-21T16:02:55.295 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: osd=1 2019-12-21T16:02:55.295 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: echo 1-42949672964 2019-12-21T16:02:55.296 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: cut -d - -f 2 2019-12-21T16:02:55.297 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: seq=42949672964 2019-12-21T16:02:55.297 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1989: flush_pg_stats: echo 'waiting osd.1 seq 42949672964' 2019-12-21T16:02:55.297 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 1 2019-12-21T16:02:55.298 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.1 seq 42949672964 2019-12-21T16:02:55.525 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 42949672964 -lt 42949672964 2019-12-21T16:02:55.525 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: flush_pg_stats: for s in $seqs 2019-12-21T16:02:55.526 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: echo 2-68719476738 2019-12-21T16:02:55.526 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: cut -d - -f 1 2019-12-21T16:02:55.527 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: osd=2 2019-12-21T16:02:55.529 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: echo 2-68719476738 2019-12-21T16:02:55.529 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: cut -d - -f 2 2019-12-21T16:02:55.529 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: seq=68719476738 2019-12-21T16:02:55.529 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1989: flush_pg_stats: echo 'waiting osd.2 seq 68719476738' 2019-12-21T16:02:55.530 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 2 2019-12-21T16:02:55.530 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.2 seq 68719476738 2019-12-21T16:02:55.758 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 68719476738 -lt 68719476738 2019-12-21T16:02:55.758 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1555: wait_for_clean: get_num_pgs 2019-12-21T16:02:55.759 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: ceph --format json status 2019-12-21T16:02:55.759 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: jq .pgmap.num_pgs 2019-12-21T16:02:56.102 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1555: wait_for_clean: test 4 == 0 2019-12-21T16:02:56.102 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1559: wait_for_clean: true 2019-12-21T16:02:56.102 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1563: wait_for_clean: get_num_active_clean 2019-12-21T16:02:56.103 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: get_num_active_clean: local expression 2019-12-21T16:02:56.103 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2019-12-21T16:02:56.103 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("stale") | not)' 2019-12-21T16:02:56.103 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: ceph --format json pg dump pgs 2019-12-21T16:02:56.103 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2019-12-21T16:02:56.325 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1563: wait_for_clean: cur_active_clean=4 2019-12-21T16:02:56.326 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1564: wait_for_clean: get_num_pgs 2019-12-21T16:02:56.326 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: ceph --format json status 2019-12-21T16:02:56.326 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: jq .pgmap.num_pgs 2019-12-21T16:02:56.595 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1564: wait_for_clean: test 4 = 4 2019-12-21T16:02:56.596 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1564: wait_for_clean: break 2019-12-21T16:02:56.596 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: return 0 2019-12-21T16:02:56.596 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:298: auto_repair_erasure_coded: create_ec_pool ecpool false k=2 m=1 2019-12-21T16:02:56.596 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: create_ec_pool: local pool_name=ecpool 2019-12-21T16:02:56.596 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2179: create_ec_pool: shift 2019-12-21T16:02:56.596 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2180: create_ec_pool: local allow_overwrites=false 2019-12-21T16:02:56.596 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: create_ec_pool: shift 2019-12-21T16:02:56.597 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=1 2019-12-21T16:02:57.053 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2019-12-21T16:02:57.053 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:539: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2019-12-21T16:02:57.573 INFO:tasks.workunit.client.0.smithi172.stderr:pool 'ecpool' created 2019-12-21T16:02:57.580 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: sleep 1 2019-12-21T16:02:58.583 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: create_ec_pool: '[' false = true ']' 2019-12-21T16:02:58.583 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2191: create_ec_pool: wait_for_clean 2019-12-21T16:02:58.583 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1548: wait_for_clean: local cmd= 2019-12-21T16:02:58.583 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1549: wait_for_clean: local num_active_clean=-1 2019-12-21T16:02:58.583 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1550: wait_for_clean: local cur_active_clean 2019-12-21T16:02:58.583 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1551: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2019-12-21T16:02:58.584 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1551: wait_for_clean: get_timeout_delays 90 .1 2019-12-21T16:02:58.584 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1494: get_timeout_delays: shopt -q -o xtrace 2019-12-21T16:02:58.585 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1494: get_timeout_delays: echo true 2019-12-21T16:02:58.585 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1494: get_timeout_delays: local trace=true 2019-12-21T16:02:58.585 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1495: get_timeout_delays: true 2019-12-21T16:02:58.585 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1495: get_timeout_delays: shopt -u -o xtrace 2019-12-21T16:02:58.688 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1551: wait_for_clean: local -a delays 2019-12-21T16:02:58.688 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1552: wait_for_clean: local -i loop=0 2019-12-21T16:02:58.688 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1554: wait_for_clean: flush_pg_stats 2019-12-21T16:02:58.688 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1977: flush_pg_stats: local timeout=300 2019-12-21T16:02:58.689 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1979: flush_pg_stats: ceph osd ls 2019-12-21T16:02:59.066 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1979: flush_pg_stats: ids='0 2019-12-21T16:02:59.066 INFO:tasks.workunit.client.0.smithi172.stderr:1 2019-12-21T16:02:59.066 INFO:tasks.workunit.client.0.smithi172.stderr:2' 2019-12-21T16:02:59.067 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1980: flush_pg_stats: seqs= 2019-12-21T16:02:59.067 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: flush_pg_stats: for osd in $ids 2019-12-21T16:02:59.067 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2019-12-21T16:02:59.190 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: seq=21474836486 2019-12-21T16:02:59.190 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: flush_pg_stats: seqs=' 0-21474836486' 2019-12-21T16:02:59.190 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: flush_pg_stats: for osd in $ids 2019-12-21T16:02:59.191 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2019-12-21T16:02:59.300 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: seq=42949672966 2019-12-21T16:02:59.300 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: flush_pg_stats: seqs=' 0-21474836486 1-42949672966' 2019-12-21T16:02:59.300 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: flush_pg_stats: for osd in $ids 2019-12-21T16:02:59.300 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2019-12-21T16:02:59.412 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: seq=68719476740 2019-12-21T16:02:59.412 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: flush_pg_stats: seqs=' 0-21474836486 1-42949672966 2-68719476740' 2019-12-21T16:02:59.413 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: flush_pg_stats: for s in $seqs 2019-12-21T16:02:59.413 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: echo 0-21474836486 2019-12-21T16:02:59.413 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: cut -d - -f 1 2019-12-21T16:02:59.414 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: osd=0 2019-12-21T16:02:59.414 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: echo 0-21474836486 2019-12-21T16:02:59.414 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: cut -d - -f 2 2019-12-21T16:02:59.415 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: seq=21474836486 2019-12-21T16:02:59.415 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1989: flush_pg_stats: echo 'waiting osd.0 seq 21474836486' 2019-12-21T16:02:59.416 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.0 seq 21474836486 2019-12-21T16:02:59.416 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 0 2019-12-21T16:02:59.649 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 21474836485 -lt 21474836486 2019-12-21T16:02:59.649 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1991: flush_pg_stats: sleep 1 2019-12-21T16:03:00.650 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1992: flush_pg_stats: '[' 300 -eq 0 ']' 2019-12-21T16:03:00.650 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 0 2019-12-21T16:03:00.888 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 21474836486 -lt 21474836486 2019-12-21T16:03:00.888 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: flush_pg_stats: for s in $seqs 2019-12-21T16:03:00.889 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: echo 1-42949672966 2019-12-21T16:03:00.889 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: cut -d - -f 1 2019-12-21T16:03:00.889 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: osd=1 2019-12-21T16:03:00.890 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: echo 1-42949672966 2019-12-21T16:03:00.890 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: cut -d - -f 2 2019-12-21T16:03:00.891 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: seq=42949672966 2019-12-21T16:03:00.891 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1989: flush_pg_stats: echo 'waiting osd.1 seq 42949672966' 2019-12-21T16:03:00.891 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 1 2019-12-21T16:03:00.891 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.1 seq 42949672966 2019-12-21T16:03:01.132 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 42949672966 -lt 42949672966 2019-12-21T16:03:01.132 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: flush_pg_stats: for s in $seqs 2019-12-21T16:03:01.133 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: echo 2-68719476740 2019-12-21T16:03:01.133 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: cut -d - -f 1 2019-12-21T16:03:01.133 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: osd=2 2019-12-21T16:03:01.134 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: echo 2-68719476740 2019-12-21T16:03:01.134 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: cut -d - -f 2 2019-12-21T16:03:01.135 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: seq=68719476740 2019-12-21T16:03:01.135 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1989: flush_pg_stats: echo 'waiting osd.2 seq 68719476740' 2019-12-21T16:03:01.135 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 2 2019-12-21T16:03:01.136 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.2 seq 68719476740 2019-12-21T16:03:01.363 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 68719476741 -lt 68719476740 2019-12-21T16:03:01.364 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1555: wait_for_clean: get_num_pgs 2019-12-21T16:03:01.364 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: ceph --format json status 2019-12-21T16:03:01.364 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: jq .pgmap.num_pgs 2019-12-21T16:03:01.634 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1555: wait_for_clean: test 5 == 0 2019-12-21T16:03:01.634 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1559: wait_for_clean: true 2019-12-21T16:03:01.635 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1563: wait_for_clean: get_num_active_clean 2019-12-21T16:03:01.635 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: get_num_active_clean: local expression 2019-12-21T16:03:01.635 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2019-12-21T16:03:01.635 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("stale") | not)' 2019-12-21T16:03:01.635 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: ceph --format json pg dump pgs 2019-12-21T16:03:01.635 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2019-12-21T16:03:01.872 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1563: wait_for_clean: cur_active_clean=5 2019-12-21T16:03:01.872 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1564: wait_for_clean: get_num_pgs 2019-12-21T16:03:01.872 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: ceph --format json status 2019-12-21T16:03:01.873 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: jq .pgmap.num_pgs 2019-12-21T16:03:02.142 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1564: wait_for_clean: test 5 = 5 2019-12-21T16:03:02.142 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1564: wait_for_clean: break 2019-12-21T16:03:02.142 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: return 0 2019-12-21T16:03:02.143 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2192: create_ec_pool: return 0 2019-12-21T16:03:02.143 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:301: auto_repair_erasure_coded: local payload=ABCDEF 2019-12-21T16:03:02.143 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:302: auto_repair_erasure_coded: echo ABCDEF 2019-12-21T16:03:02.143 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:303: auto_repair_erasure_coded: rados --pool ecpool put SOMETHING td/osd-scrub-repair/ORIGINAL 2019-12-21T16:03:02.233 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:307: auto_repair_erasure_coded: get_not_primary ecpool SOMETHING 2019-12-21T16:03:02.233 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1165: get_not_primary: local poolname=ecpool 2019-12-21T16:03:02.233 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1166: get_not_primary: local objectname=SOMETHING 2019-12-21T16:03:02.233 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: get_not_primary: get_primary ecpool SOMETHING 2019-12-21T16:03:02.233 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_primary: local poolname=ecpool 2019-12-21T16:03:02.234 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1133: get_primary: local objectname=SOMETHING 2019-12-21T16:03:02.234 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1135: get_primary: ceph --format json osd map ecpool SOMETHING 2019-12-21T16:03:02.234 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1136: get_primary: jq .acting_primary 2019-12-21T16:03:02.465 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: get_not_primary: local primary=2 2019-12-21T16:03:02.465 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1169: get_not_primary: ceph --format json osd map ecpool SOMETHING 2019-12-21T16:03:02.465 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1170: get_not_primary: jq '.acting | map(select (. != 2)) | .[0]' 2019-12-21T16:03:02.743 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:307: auto_repair_erasure_coded: objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2019-12-21T16:03:02.744 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: objectstore_tool: local dir=td/osd-scrub-repair 2019-12-21T16:03:02.744 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: objectstore_tool: shift 2019-12-21T16:03:02.744 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1235: objectstore_tool: local id=1 2019-12-21T16:03:02.744 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1236: objectstore_tool: shift 2019-12-21T16:03:02.744 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1238: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING remove 2019-12-21T16:03:02.744 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1205: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2019-12-21T16:03:02.745 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nowait: shift 2019-12-21T16:03:02.745 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1207: _objectstore_tool_nowait: local id=1 2019-12-21T16:03:02.745 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1208: _objectstore_tool_nowait: shift 2019-12-21T16:03:02.745 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1210: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2019-12-21T16:03:02.745 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2019-12-21T16:03:02.745 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2019-12-21T16:03:02.745 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2019-12-21T16:03:02.746 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2019-12-21T16:03:02.746 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2019-12-21T16:03:02.849 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2019-12-21T16:03:02.850 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1212: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING remove 2019-12-21T16:03:02.850 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1193: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2019-12-21T16:03:02.850 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1194: _objectstore_tool_nodown: shift 2019-12-21T16:03:02.850 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1195: _objectstore_tool_nodown: local id=1 2019-12-21T16:03:02.850 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: _objectstore_tool_nodown: shift 2019-12-21T16:03:02.850 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2019-12-21T16:03:02.850 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING remove 2019-12-21T16:03:03.230 INFO:tasks.workunit.client.0.smithi172.stderr:Error getting attr on : 1.3_head,#-3:c0000000:::scrub_1.3:head#, (61) No data available 2019-12-21T16:03:03.231 INFO:tasks.workunit.client.0.smithi172.stderr:Error getting attr on : 1.0_head,#-3:00000000:::scrub_1.0:head#, (61) No data available 2019-12-21T16:03:03.231 INFO:tasks.workunit.client.0.smithi172.stdout:remove 1#2:eb822e21:::SOMETHING:head# 2019-12-21T16:03:03.793 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1213: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2019-12-21T16:03:03.794 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:842: activate_osd: local dir=td/osd-scrub-repair 2019-12-21T16:03:03.794 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: shift 2019-12-21T16:03:03.794 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: local id=1 2019-12-21T16:03:03.794 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: shift 2019-12-21T16:03:03.794 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local osd_data=td/osd-scrub-repair/1 2019-12-21T16:03:03.794 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local 'ceph_args=--fsid=973e3fc1-e63e-4bde-9db6-84c68fd5e93e --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2019-12-21T16:03:03.794 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2019-12-21T16:03:03.795 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-journal-size=100' 2019-12-21T16:03:03.795 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2019-12-21T16:03:03.795 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2019-12-21T16:03:03.795 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2019-12-21T16:03:03.795 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --chdir=' 2019-12-21T16:03:03.795 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+= 2019-12-21T16:03:03.795 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2019-12-21T16:03:03.796 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: get_asok_path 2019-12-21T16:03:03.796 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2019-12-21T16:03:03.796 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2019-12-21T16:03:03.796 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2019-12-21T16:03:03.796 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T16:03:03.796 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.119575 2019-12-21T16:03:03.796 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.119575/$cluster-$name.asok' 2019-12-21T16:03:03.797 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.119575/$cluster-$name.asok' 2019-12-21T16:03:03.797 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --debug-osd=20' 2019-12-21T16:03:03.797 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2019-12-21T16:03:03.797 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2019-12-21T16:03:03.797 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2019-12-21T16:03:03.797 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2019-12-21T16:03:03.797 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2019-12-21T16:03:03.798 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' ' 2019-12-21T16:03:03.798 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2019-12-21T16:03:03.798 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: mkdir -p td/osd-scrub-repair/1 2019-12-21T16:03:03.798 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: echo start osd.1 2019-12-21T16:03:03.798 INFO:tasks.workunit.client.0.smithi172.stderr:start osd.1 2019-12-21T16:03:03.798 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph-osd -i 1 --fsid=973e3fc1-e63e-4bde-9db6-84c68fd5e93e --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.119575/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2019-12-21T16:03:03.798 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: cat td/osd-scrub-repair/1/whoami 2019-12-21T16:03:03.799 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: '[' 1 = 1 ']' 2019-12-21T16:03:03.799 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph osd dump --format=json 2019-12-21T16:03:03.801 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: jq '.flags_set[]' 2019-12-21T16:03:03.801 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: grep -q '"noup"' 2019-12-21T16:03:03.816 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:03:03.814+0000 7f3fc4deeec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T16:03:03.817 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:03:03.816+0000 7f3fc4deeec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T16:03:03.820 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:03:03.818+0000 7f3fc4deeec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T16:03:03.821 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:03:03.820+0000 7f3fc4deeec0 -1 Falling back to public interface 2019-12-21T16:03:04.028 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: wait_for_osd up 1 2019-12-21T16:03:04.028 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:914: wait_for_osd: local state=up 2019-12-21T16:03:04.028 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local id=1 2019-12-21T16:03:04.028 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: status=1 2019-12-21T16:03:04.028 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i=0 )) 2019-12-21T16:03:04.029 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i < 300 )) 2019-12-21T16:03:04.029 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: echo 0 2019-12-21T16:03:04.029 INFO:tasks.workunit.client.0.smithi172.stderr:0 2019-12-21T16:03:04.029 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: ceph osd dump 2019-12-21T16:03:04.029 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: grep 'osd.1 up' 2019-12-21T16:03:04.255 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: sleep 1 2019-12-21T16:03:04.343 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:03:04.341+0000 7f3fc4deeec0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2019-12-21T16:03:04.350 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:03:04.349+0000 7f3fc4deeec0 -1 journal do_read_entry(413696): bad header magic 2019-12-21T16:03:04.351 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:03:04.349+0000 7f3fc4deeec0 -1 journal do_read_entry(413696): bad header magic 2019-12-21T16:03:04.357 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:03:04.355+0000 7f3fc4deeec0 -1 osd.1 37 log_to_monitors {default=true} 2019-12-21T16:03:05.256 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i++ )) 2019-12-21T16:03:05.256 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i < 300 )) 2019-12-21T16:03:05.256 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: echo 1 2019-12-21T16:03:05.257 INFO:tasks.workunit.client.0.smithi172.stderr:1 2019-12-21T16:03:05.257 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: ceph osd dump 2019-12-21T16:03:05.257 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: grep 'osd.1 up' 2019-12-21T16:03:05.486 INFO:tasks.workunit.client.0.smithi172.stderr:osd.1 up in weight 1 up_from 42 up_thru 42 down_at 38 last_clean_interval [10,37) [v2:127.0.0.1:6810/146869,v1:127.0.0.1:6811/146869] [v2:127.0.0.1:6812/146869,v1:127.0.0.1:6813/146869] exists,up 46c8d16e-77f0-4445-965e-c865f6694611 2019-12-21T16:03:05.486 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=0 2019-12-21T16:03:05.486 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: break 2019-12-21T16:03:05.486 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: return 0 2019-12-21T16:03:05.487 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1239: objectstore_tool: wait_for_clean 2019-12-21T16:03:05.487 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1548: wait_for_clean: local cmd= 2019-12-21T16:03:05.487 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1549: wait_for_clean: local num_active_clean=-1 2019-12-21T16:03:05.487 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1550: wait_for_clean: local cur_active_clean 2019-12-21T16:03:05.487 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1551: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2019-12-21T16:03:05.487 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1551: wait_for_clean: get_timeout_delays 90 .1 2019-12-21T16:03:05.487 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1494: get_timeout_delays: shopt -q -o xtrace 2019-12-21T16:03:05.488 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1494: get_timeout_delays: echo true 2019-12-21T16:03:05.488 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1494: get_timeout_delays: local trace=true 2019-12-21T16:03:05.488 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1495: get_timeout_delays: true 2019-12-21T16:03:05.488 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1495: get_timeout_delays: shopt -u -o xtrace 2019-12-21T16:03:05.578 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1551: wait_for_clean: local -a delays 2019-12-21T16:03:05.579 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1552: wait_for_clean: local -i loop=0 2019-12-21T16:03:05.579 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1554: wait_for_clean: flush_pg_stats 2019-12-21T16:03:05.579 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1977: flush_pg_stats: local timeout=300 2019-12-21T16:03:05.579 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1979: flush_pg_stats: ceph osd ls 2019-12-21T16:03:05.805 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1979: flush_pg_stats: ids='0 2019-12-21T16:03:05.805 INFO:tasks.workunit.client.0.smithi172.stderr:1 2019-12-21T16:03:05.805 INFO:tasks.workunit.client.0.smithi172.stderr:2' 2019-12-21T16:03:05.805 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1980: flush_pg_stats: seqs= 2019-12-21T16:03:05.806 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: flush_pg_stats: for osd in $ids 2019-12-21T16:03:05.806 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2019-12-21T16:03:05.914 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: seq=21474836488 2019-12-21T16:03:05.914 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: flush_pg_stats: seqs=' 0-21474836488' 2019-12-21T16:03:05.914 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: flush_pg_stats: for osd in $ids 2019-12-21T16:03:05.915 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2019-12-21T16:03:06.022 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: seq=180388626434 2019-12-21T16:03:06.022 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: flush_pg_stats: seqs=' 0-21474836488 1-180388626434' 2019-12-21T16:03:06.023 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: flush_pg_stats: for osd in $ids 2019-12-21T16:03:06.023 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2019-12-21T16:03:06.132 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: seq=68719476743 2019-12-21T16:03:06.132 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: flush_pg_stats: seqs=' 0-21474836488 1-180388626434 2-68719476743' 2019-12-21T16:03:06.132 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: flush_pg_stats: for s in $seqs 2019-12-21T16:03:06.133 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: echo 0-21474836488 2019-12-21T16:03:06.133 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: cut -d - -f 1 2019-12-21T16:03:06.133 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: osd=0 2019-12-21T16:03:06.134 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: echo 0-21474836488 2019-12-21T16:03:06.134 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: cut -d - -f 2 2019-12-21T16:03:06.135 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: seq=21474836488 2019-12-21T16:03:06.135 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1989: flush_pg_stats: echo 'waiting osd.0 seq 21474836488' 2019-12-21T16:03:06.136 INFO:tasks.workunit.client.0.smithi172.stderr:waiting osd.0 seq 21474836488 2019-12-21T16:03:06.136 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 0 2019-12-21T16:03:06.362 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 21474836487 -lt 21474836488 2019-12-21T16:03:06.362 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1991: flush_pg_stats: sleep 1 2019-12-21T16:03:07.363 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1992: flush_pg_stats: '[' 300 -eq 0 ']' 2019-12-21T16:03:07.363 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 0 2019-12-21T16:03:07.592 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 21474836488 -lt 21474836488 2019-12-21T16:03:07.592 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: flush_pg_stats: for s in $seqs 2019-12-21T16:03:07.593 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: echo 1-180388626434 2019-12-21T16:03:07.593 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: cut -d - -f 1 2019-12-21T16:03:07.593 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: osd=1 2019-12-21T16:03:07.594 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: echo 1-180388626434 2019-12-21T16:03:07.594 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: cut -d - -f 2 2019-12-21T16:03:07.595 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: seq=180388626434 2019-12-21T16:03:07.595 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1989: flush_pg_stats: echo 'waiting osd.1 seq 180388626434' 2019-12-21T16:03:07.595 INFO:tasks.workunit.client.0.smithi172.stderr:waiting osd.1 seq 180388626434 2019-12-21T16:03:07.596 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 1 2019-12-21T16:03:07.824 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 180388626434 -lt 180388626434 2019-12-21T16:03:07.824 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: flush_pg_stats: for s in $seqs 2019-12-21T16:03:07.825 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: echo 2-68719476743 2019-12-21T16:03:07.825 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: cut -d - -f 1 2019-12-21T16:03:07.826 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: osd=2 2019-12-21T16:03:07.826 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: echo 2-68719476743 2019-12-21T16:03:07.826 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: cut -d - -f 2 2019-12-21T16:03:07.827 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: seq=68719476743 2019-12-21T16:03:07.828 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1989: flush_pg_stats: echo 'waiting osd.2 seq 68719476743' 2019-12-21T16:03:07.828 INFO:tasks.workunit.client.0.smithi172.stderr:waiting osd.2 seq 68719476743 2019-12-21T16:03:07.828 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 2 2019-12-21T16:03:08.068 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 68719476743 -lt 68719476743 2019-12-21T16:03:08.068 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1555: wait_for_clean: get_num_pgs 2019-12-21T16:03:08.069 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: ceph --format json status 2019-12-21T16:03:08.069 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: jq .pgmap.num_pgs 2019-12-21T16:03:08.336 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1555: wait_for_clean: test 5 == 0 2019-12-21T16:03:08.337 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1559: wait_for_clean: true 2019-12-21T16:03:08.337 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1563: wait_for_clean: get_num_active_clean 2019-12-21T16:03:08.337 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: get_num_active_clean: local expression 2019-12-21T16:03:08.337 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2019-12-21T16:03:08.337 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("stale") | not)' 2019-12-21T16:03:08.338 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: ceph --format json pg dump pgs 2019-12-21T16:03:08.338 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2019-12-21T16:03:08.558 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1563: wait_for_clean: cur_active_clean=5 2019-12-21T16:03:08.558 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1564: wait_for_clean: get_num_pgs 2019-12-21T16:03:08.559 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: ceph --format json status 2019-12-21T16:03:08.559 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: jq .pgmap.num_pgs 2019-12-21T16:03:08.830 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1564: wait_for_clean: test 5 = 5 2019-12-21T16:03:08.831 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1564: wait_for_clean: break 2019-12-21T16:03:08.831 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: return 0 2019-12-21T16:03:08.831 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:309: auto_repair_erasure_coded: get_pg ecpool SOMETHING 2019-12-21T16:03:08.831 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1029: get_pg: local poolname=ecpool 2019-12-21T16:03:08.831 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1030: get_pg: local objectname=SOMETHING 2019-12-21T16:03:08.831 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1032: get_pg: ceph --format json osd map ecpool SOMETHING 2019-12-21T16:03:08.831 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1032: get_pg: jq -r .pgid 2019-12-21T16:03:09.091 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:309: auto_repair_erasure_coded: local pgid=2.0 2019-12-21T16:03:09.092 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:310: auto_repair_erasure_coded: get_last_scrub_stamp 2.0 2019-12-21T16:03:09.092 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=2.0 2019-12-21T16:03:09.092 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T16:03:09.092 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T16:03:09.093 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2019-12-21T16:03:09.317 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:310: auto_repair_erasure_coded: wait_for_scrub 2.0 2019-12-21T16:02:57.492990+0000 2019-12-21T16:03:09.318 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1789: wait_for_scrub: local pgid=2.0 2019-12-21T16:03:09.318 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1790: wait_for_scrub: local last_scrub=2019-12-21T16:02:57.492990+0000 2019-12-21T16:03:09.318 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1791: wait_for_scrub: local sname=last_scrub_stamp 2019-12-21T16:03:09.318 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i=0 )) 2019-12-21T16:03:09.318 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T16:03:09.319 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2019-12-21T16:03:09.319 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=2.0 2019-12-21T16:03:09.319 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T16:03:09.319 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T16:03:09.319 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2019-12-21T16:03:09.543 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T16:02:57.492990+0000 '>' 2019-12-21T16:02:57.492990+0000 2019-12-21T16:03:09.544 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T16:03:10.544 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T16:03:10.545 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T16:03:10.545 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2019-12-21T16:03:10.546 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=2.0 2019-12-21T16:03:10.546 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T16:03:10.546 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T16:03:10.546 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2019-12-21T16:03:10.778 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T16:03:09.479604+0000 '>' 2019-12-21T16:02:57.492990+0000 2019-12-21T16:03:10.778 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1795: wait_for_scrub: return 0 2019-12-21T16:03:10.778 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:311: auto_repair_erasure_coded: wait_for_clean 2019-12-21T16:03:10.778 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1548: wait_for_clean: local cmd= 2019-12-21T16:03:10.779 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1549: wait_for_clean: local num_active_clean=-1 2019-12-21T16:03:10.779 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1550: wait_for_clean: local cur_active_clean 2019-12-21T16:03:10.779 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1551: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2019-12-21T16:03:10.779 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1551: wait_for_clean: get_timeout_delays 90 .1 2019-12-21T16:03:10.779 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1494: get_timeout_delays: shopt -q -o xtrace 2019-12-21T16:03:10.779 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1494: get_timeout_delays: echo true 2019-12-21T16:03:10.779 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1494: get_timeout_delays: local trace=true 2019-12-21T16:03:10.780 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1495: get_timeout_delays: true 2019-12-21T16:03:10.780 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1495: get_timeout_delays: shopt -u -o xtrace 2019-12-21T16:03:10.871 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1551: wait_for_clean: local -a delays 2019-12-21T16:03:10.871 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1552: wait_for_clean: local -i loop=0 2019-12-21T16:03:10.871 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1554: wait_for_clean: flush_pg_stats 2019-12-21T16:03:10.871 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1977: flush_pg_stats: local timeout=300 2019-12-21T16:03:10.872 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1979: flush_pg_stats: ceph osd ls 2019-12-21T16:03:11.098 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1979: flush_pg_stats: ids='0 2019-12-21T16:03:11.098 INFO:tasks.workunit.client.0.smithi172.stderr:1 2019-12-21T16:03:11.098 INFO:tasks.workunit.client.0.smithi172.stderr:2' 2019-12-21T16:03:11.099 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1980: flush_pg_stats: seqs= 2019-12-21T16:03:11.099 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: flush_pg_stats: for osd in $ids 2019-12-21T16:03:11.099 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2019-12-21T16:03:11.208 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: seq=21474836490 2019-12-21T16:03:11.208 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: flush_pg_stats: seqs=' 0-21474836490' 2019-12-21T16:03:11.208 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: flush_pg_stats: for osd in $ids 2019-12-21T16:03:11.208 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2019-12-21T16:03:11.316 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: seq=180388626436 2019-12-21T16:03:11.316 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: flush_pg_stats: seqs=' 0-21474836490 1-180388626436' 2019-12-21T16:03:11.317 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: flush_pg_stats: for osd in $ids 2019-12-21T16:03:11.317 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2019-12-21T16:03:11.425 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: seq=68719476745 2019-12-21T16:03:11.425 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: flush_pg_stats: seqs=' 0-21474836490 1-180388626436 2-68719476745' 2019-12-21T16:03:11.425 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: flush_pg_stats: for s in $seqs 2019-12-21T16:03:11.426 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: echo 0-21474836490 2019-12-21T16:03:11.426 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: cut -d - -f 1 2019-12-21T16:03:11.427 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: osd=0 2019-12-21T16:03:11.427 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: echo 0-21474836490 2019-12-21T16:03:11.427 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: cut -d - -f 2 2019-12-21T16:03:11.428 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: seq=21474836490 2019-12-21T16:03:11.428 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1989: flush_pg_stats: echo 'waiting osd.0 seq 21474836490' 2019-12-21T16:03:11.428 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 0 2019-12-21T16:03:11.428 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.0 seq 21474836490 2019-12-21T16:03:11.673 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 21474836489 -lt 21474836490 2019-12-21T16:03:11.673 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1991: flush_pg_stats: sleep 1 2019-12-21T16:03:12.674 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1992: flush_pg_stats: '[' 300 -eq 0 ']' 2019-12-21T16:03:12.675 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 0 2019-12-21T16:03:12.919 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 21474836490 -lt 21474836490 2019-12-21T16:03:12.919 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: flush_pg_stats: for s in $seqs 2019-12-21T16:03:12.920 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: echo 1-180388626436 2019-12-21T16:03:12.920 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: cut -d - -f 1 2019-12-21T16:03:12.920 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: osd=1 2019-12-21T16:03:12.921 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: echo 1-180388626436 2019-12-21T16:03:12.921 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: cut -d - -f 2 2019-12-21T16:03:12.922 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: seq=180388626436 2019-12-21T16:03:12.923 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1989: flush_pg_stats: echo 'waiting osd.1 seq 180388626436' 2019-12-21T16:03:12.923 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 1 2019-12-21T16:03:12.923 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.1 seq 180388626436 2019-12-21T16:03:13.149 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 180388626436 -lt 180388626436 2019-12-21T16:03:13.150 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: flush_pg_stats: for s in $seqs 2019-12-21T16:03:13.150 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: echo 2-68719476745 2019-12-21T16:03:13.150 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: cut -d - -f 1 2019-12-21T16:03:13.151 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: osd=2 2019-12-21T16:03:13.151 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: echo 2-68719476745 2019-12-21T16:03:13.152 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: cut -d - -f 2 2019-12-21T16:03:13.153 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: seq=68719476745 2019-12-21T16:03:13.153 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1989: flush_pg_stats: echo 'waiting osd.2 seq 68719476745' 2019-12-21T16:03:13.153 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 2 2019-12-21T16:03:13.153 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.2 seq 68719476745 2019-12-21T16:03:13.445 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 68719476745 -lt 68719476745 2019-12-21T16:03:13.446 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1555: wait_for_clean: get_num_pgs 2019-12-21T16:03:13.446 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: ceph --format json status 2019-12-21T16:03:13.446 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: jq .pgmap.num_pgs 2019-12-21T16:03:13.713 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1555: wait_for_clean: test 5 == 0 2019-12-21T16:03:13.713 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1559: wait_for_clean: true 2019-12-21T16:03:13.714 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1563: wait_for_clean: get_num_active_clean 2019-12-21T16:03:13.714 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: get_num_active_clean: local expression 2019-12-21T16:03:13.714 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2019-12-21T16:03:13.714 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("stale") | not)' 2019-12-21T16:03:13.714 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: ceph --format json pg dump pgs 2019-12-21T16:03:13.714 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2019-12-21T16:03:13.936 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1563: wait_for_clean: cur_active_clean=5 2019-12-21T16:03:13.936 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1564: wait_for_clean: get_num_pgs 2019-12-21T16:03:13.937 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: ceph --format json status 2019-12-21T16:03:13.937 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: jq .pgmap.num_pgs 2019-12-21T16:03:14.230 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1564: wait_for_clean: test 5 = 5 2019-12-21T16:03:14.230 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1564: wait_for_clean: break 2019-12-21T16:03:14.230 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: return 0 2019-12-21T16:03:14.230 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:314: auto_repair_erasure_coded: get_not_primary ecpool SOMETHING 2019-12-21T16:03:14.231 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1165: get_not_primary: local poolname=ecpool 2019-12-21T16:03:14.231 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1166: get_not_primary: local objectname=SOMETHING 2019-12-21T16:03:14.231 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: get_not_primary: get_primary ecpool SOMETHING 2019-12-21T16:03:14.231 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_primary: local poolname=ecpool 2019-12-21T16:03:14.231 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1133: get_primary: local objectname=SOMETHING 2019-12-21T16:03:14.231 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1135: get_primary: ceph --format json osd map ecpool SOMETHING 2019-12-21T16:03:14.231 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1136: get_primary: jq .acting_primary 2019-12-21T16:03:14.474 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: get_not_primary: local primary=2 2019-12-21T16:03:14.474 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1169: get_not_primary: ceph --format json osd map ecpool SOMETHING 2019-12-21T16:03:14.474 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1170: get_not_primary: jq '.acting | map(select (. != 2)) | .[0]' 2019-12-21T16:03:14.706 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:314: auto_repair_erasure_coded: objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2019-12-21T16:03:14.706 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: objectstore_tool: local dir=td/osd-scrub-repair 2019-12-21T16:03:14.706 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: objectstore_tool: shift 2019-12-21T16:03:14.706 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1235: objectstore_tool: local id=1 2019-12-21T16:03:14.707 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1236: objectstore_tool: shift 2019-12-21T16:03:14.707 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1238: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING list-attrs 2019-12-21T16:03:14.707 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1205: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2019-12-21T16:03:14.707 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nowait: shift 2019-12-21T16:03:14.707 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1207: _objectstore_tool_nowait: local id=1 2019-12-21T16:03:14.707 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1208: _objectstore_tool_nowait: shift 2019-12-21T16:03:14.707 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1210: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2019-12-21T16:03:14.708 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2019-12-21T16:03:14.708 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2019-12-21T16:03:14.708 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2019-12-21T16:03:14.708 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2019-12-21T16:03:14.708 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2019-12-21T16:03:14.811 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2019-12-21T16:03:14.812 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1212: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING list-attrs 2019-12-21T16:03:14.812 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1193: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2019-12-21T16:03:14.812 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1194: _objectstore_tool_nodown: shift 2019-12-21T16:03:14.813 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1195: _objectstore_tool_nodown: local id=1 2019-12-21T16:03:14.813 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: _objectstore_tool_nodown: shift 2019-12-21T16:03:14.813 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2019-12-21T16:03:14.813 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING list-attrs 2019-12-21T16:03:15.372 INFO:tasks.workunit.client.0.smithi172.stderr:Error getting attr on : 1.3_head,#-3:c0000000:::scrub_1.3:head#, (61) No data available 2019-12-21T16:03:15.372 INFO:tasks.workunit.client.0.smithi172.stderr:Error getting attr on : 1.0_head,#-3:00000000:::scrub_1.0:head#, (61) No data available 2019-12-21T16:03:15.373 INFO:tasks.workunit.client.0.smithi172.stdout:_ 2019-12-21T16:03:15.373 INFO:tasks.workunit.client.0.smithi172.stdout:hinfo_key 2019-12-21T16:03:15.373 INFO:tasks.workunit.client.0.smithi172.stdout:snapset 2019-12-21T16:03:15.749 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1213: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2019-12-21T16:03:15.749 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:842: activate_osd: local dir=td/osd-scrub-repair 2019-12-21T16:03:15.749 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: shift 2019-12-21T16:03:15.749 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: local id=1 2019-12-21T16:03:15.749 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: shift 2019-12-21T16:03:15.750 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local osd_data=td/osd-scrub-repair/1 2019-12-21T16:03:15.750 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local 'ceph_args=--fsid=973e3fc1-e63e-4bde-9db6-84c68fd5e93e --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2019-12-21T16:03:15.750 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2019-12-21T16:03:15.750 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-journal-size=100' 2019-12-21T16:03:15.750 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2019-12-21T16:03:15.751 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2019-12-21T16:03:15.751 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2019-12-21T16:03:15.751 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --chdir=' 2019-12-21T16:03:15.751 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+= 2019-12-21T16:03:15.751 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2019-12-21T16:03:15.751 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: get_asok_path 2019-12-21T16:03:15.752 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2019-12-21T16:03:15.752 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2019-12-21T16:03:15.752 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2019-12-21T16:03:15.752 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T16:03:15.752 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.119575 2019-12-21T16:03:15.752 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.119575/$cluster-$name.asok' 2019-12-21T16:03:15.752 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.119575/$cluster-$name.asok' 2019-12-21T16:03:15.753 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --debug-osd=20' 2019-12-21T16:03:15.753 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2019-12-21T16:03:15.753 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2019-12-21T16:03:15.753 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2019-12-21T16:03:15.753 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2019-12-21T16:03:15.753 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2019-12-21T16:03:15.753 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' ' 2019-12-21T16:03:15.754 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2019-12-21T16:03:15.754 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: mkdir -p td/osd-scrub-repair/1 2019-12-21T16:03:15.754 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: echo start osd.1 2019-12-21T16:03:15.754 INFO:tasks.workunit.client.0.smithi172.stderr:start osd.1 2019-12-21T16:03:15.754 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph-osd -i 1 --fsid=973e3fc1-e63e-4bde-9db6-84c68fd5e93e --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.119575/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2019-12-21T16:03:15.754 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: cat td/osd-scrub-repair/1/whoami 2019-12-21T16:03:15.754 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: '[' 1 = 1 ']' 2019-12-21T16:03:15.755 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph osd dump --format=json 2019-12-21T16:03:15.755 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: jq '.flags_set[]' 2019-12-21T16:03:15.755 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: grep -q '"noup"' 2019-12-21T16:03:15.770 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:03:15.768+0000 7fdb0e01eec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T16:03:15.852 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:03:15.849+0000 7fdb0e01eec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T16:03:15.853 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:03:15.850+0000 7fdb0e01eec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T16:03:15.854 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:03:15.852+0000 7fdb0e01eec0 -1 Falling back to public interface 2019-12-21T16:03:16.000 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: wait_for_osd up 1 2019-12-21T16:03:16.000 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:914: wait_for_osd: local state=up 2019-12-21T16:03:16.000 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local id=1 2019-12-21T16:03:16.000 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: status=1 2019-12-21T16:03:16.001 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i=0 )) 2019-12-21T16:03:16.001 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i < 300 )) 2019-12-21T16:03:16.002 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: echo 0 2019-12-21T16:03:16.002 INFO:tasks.workunit.client.0.smithi172.stderr:0 2019-12-21T16:03:16.002 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: ceph osd dump 2019-12-21T16:03:16.002 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: grep 'osd.1 up' 2019-12-21T16:03:16.302 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: sleep 1 2019-12-21T16:03:16.361 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:03:16.358+0000 7fdb0e01eec0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2019-12-21T16:03:16.361 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:03:16.358+0000 7fdb0e01eec0 -1 journal do_read_entry(679936): bad header magic 2019-12-21T16:03:16.361 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:03:16.358+0000 7fdb0e01eec0 -1 journal do_read_entry(679936): bad header magic 2019-12-21T16:03:16.367 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:03:16.365+0000 7fdb0e01eec0 -1 osd.1 48 log_to_monitors {default=true} 2019-12-21T16:03:17.304 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i++ )) 2019-12-21T16:03:17.304 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i < 300 )) 2019-12-21T16:03:17.305 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: echo 1 2019-12-21T16:03:17.305 INFO:tasks.workunit.client.0.smithi172.stderr:1 2019-12-21T16:03:17.305 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: ceph osd dump 2019-12-21T16:03:17.305 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: grep 'osd.1 up' 2019-12-21T16:03:17.577 INFO:tasks.workunit.client.0.smithi172.stderr:osd.1 up in weight 1 up_from 53 up_thru 53 down_at 49 last_clean_interval [42,48) [v2:127.0.0.1:6810/148089,v1:127.0.0.1:6811/148089] [v2:127.0.0.1:6812/148089,v1:127.0.0.1:6813/148089] exists,up 46c8d16e-77f0-4445-965e-c865f6694611 2019-12-21T16:03:17.577 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=0 2019-12-21T16:03:17.577 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: break 2019-12-21T16:03:17.578 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: return 0 2019-12-21T16:03:17.578 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1239: objectstore_tool: wait_for_clean 2019-12-21T16:03:17.578 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1548: wait_for_clean: local cmd= 2019-12-21T16:03:17.578 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1549: wait_for_clean: local num_active_clean=-1 2019-12-21T16:03:17.578 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1550: wait_for_clean: local cur_active_clean 2019-12-21T16:03:17.578 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1551: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2019-12-21T16:03:17.578 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1551: wait_for_clean: get_timeout_delays 90 .1 2019-12-21T16:03:17.579 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1494: get_timeout_delays: shopt -q -o xtrace 2019-12-21T16:03:17.579 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1494: get_timeout_delays: echo true 2019-12-21T16:03:17.579 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1494: get_timeout_delays: local trace=true 2019-12-21T16:03:17.579 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1495: get_timeout_delays: true 2019-12-21T16:03:17.579 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1495: get_timeout_delays: shopt -u -o xtrace 2019-12-21T16:03:17.670 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1551: wait_for_clean: local -a delays 2019-12-21T16:03:17.670 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1552: wait_for_clean: local -i loop=0 2019-12-21T16:03:17.670 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1554: wait_for_clean: flush_pg_stats 2019-12-21T16:03:17.670 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1977: flush_pg_stats: local timeout=300 2019-12-21T16:03:17.670 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1979: flush_pg_stats: ceph osd ls 2019-12-21T16:03:17.901 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1979: flush_pg_stats: ids='0 2019-12-21T16:03:17.902 INFO:tasks.workunit.client.0.smithi172.stderr:1 2019-12-21T16:03:17.902 INFO:tasks.workunit.client.0.smithi172.stderr:2' 2019-12-21T16:03:17.902 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1980: flush_pg_stats: seqs= 2019-12-21T16:03:17.902 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: flush_pg_stats: for osd in $ids 2019-12-21T16:03:17.902 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2019-12-21T16:03:18.011 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: seq=21474836493 2019-12-21T16:03:18.011 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: flush_pg_stats: seqs=' 0-21474836493' 2019-12-21T16:03:18.011 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: flush_pg_stats: for osd in $ids 2019-12-21T16:03:18.012 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2019-12-21T16:03:18.120 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: seq=227633266690 2019-12-21T16:03:18.120 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: flush_pg_stats: seqs=' 0-21474836493 1-227633266690' 2019-12-21T16:03:18.120 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: flush_pg_stats: for osd in $ids 2019-12-21T16:03:18.120 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2019-12-21T16:03:18.231 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: seq=68719476747 2019-12-21T16:03:18.231 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: flush_pg_stats: seqs=' 0-21474836493 1-227633266690 2-68719476747' 2019-12-21T16:03:18.232 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: flush_pg_stats: for s in $seqs 2019-12-21T16:03:18.232 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: echo 0-21474836493 2019-12-21T16:03:18.232 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: cut -d - -f 1 2019-12-21T16:03:18.232 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: osd=0 2019-12-21T16:03:18.233 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: echo 0-21474836493 2019-12-21T16:03:18.234 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: cut -d - -f 2 2019-12-21T16:03:18.234 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: seq=21474836493 2019-12-21T16:03:18.234 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1989: flush_pg_stats: echo 'waiting osd.0 seq 21474836493' 2019-12-21T16:03:18.234 INFO:tasks.workunit.client.0.smithi172.stderr:waiting osd.0 seq 21474836493 2019-12-21T16:03:18.234 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 0 2019-12-21T16:03:18.460 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 21474836491 -lt 21474836493 2019-12-21T16:03:18.460 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1991: flush_pg_stats: sleep 1 2019-12-21T16:03:19.461 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1992: flush_pg_stats: '[' 300 -eq 0 ']' 2019-12-21T16:03:19.462 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 0 2019-12-21T16:03:19.693 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 21474836493 -lt 21474836493 2019-12-21T16:03:19.694 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: flush_pg_stats: for s in $seqs 2019-12-21T16:03:19.694 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: echo 1-227633266690 2019-12-21T16:03:19.694 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: cut -d - -f 1 2019-12-21T16:03:19.695 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: osd=1 2019-12-21T16:03:19.695 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: echo 1-227633266690 2019-12-21T16:03:19.695 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: cut -d - -f 2 2019-12-21T16:03:19.696 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: seq=227633266690 2019-12-21T16:03:19.696 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1989: flush_pg_stats: echo 'waiting osd.1 seq 227633266690' 2019-12-21T16:03:19.697 INFO:tasks.workunit.client.0.smithi172.stderr:waiting osd.1 seq 227633266690 2019-12-21T16:03:19.697 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 1 2019-12-21T16:03:19.923 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 227633266690 -lt 227633266690 2019-12-21T16:03:19.923 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: flush_pg_stats: for s in $seqs 2019-12-21T16:03:19.923 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: echo 2-68719476747 2019-12-21T16:03:19.924 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: cut -d - -f 1 2019-12-21T16:03:19.924 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: osd=2 2019-12-21T16:03:19.925 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: echo 2-68719476747 2019-12-21T16:03:19.925 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: cut -d - -f 2 2019-12-21T16:03:19.926 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: seq=68719476747 2019-12-21T16:03:19.926 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1989: flush_pg_stats: echo 'waiting osd.2 seq 68719476747' 2019-12-21T16:03:19.926 INFO:tasks.workunit.client.0.smithi172.stderr:waiting osd.2 seq 68719476747 2019-12-21T16:03:19.926 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 2 2019-12-21T16:03:20.239 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 68719476747 -lt 68719476747 2019-12-21T16:03:20.239 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1555: wait_for_clean: get_num_pgs 2019-12-21T16:03:20.240 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: ceph --format json status 2019-12-21T16:03:20.240 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: jq .pgmap.num_pgs 2019-12-21T16:03:20.555 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1555: wait_for_clean: test 5 == 0 2019-12-21T16:03:20.555 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1559: wait_for_clean: true 2019-12-21T16:03:20.556 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1563: wait_for_clean: get_num_active_clean 2019-12-21T16:03:20.556 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: get_num_active_clean: local expression 2019-12-21T16:03:20.556 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2019-12-21T16:03:20.556 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("stale") | not)' 2019-12-21T16:03:20.556 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: ceph --format json pg dump pgs 2019-12-21T16:03:20.556 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2019-12-21T16:03:20.779 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1563: wait_for_clean: cur_active_clean=5 2019-12-21T16:03:20.780 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1564: wait_for_clean: get_num_pgs 2019-12-21T16:03:20.780 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: ceph --format json status 2019-12-21T16:03:20.780 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: jq .pgmap.num_pgs 2019-12-21T16:03:21.050 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1564: wait_for_clean: test 5 = 5 2019-12-21T16:03:21.050 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1564: wait_for_clean: break 2019-12-21T16:03:21.050 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: return 0 2019-12-21T16:03:21.050 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:315: auto_repair_erasure_coded: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2019-12-21T16:03:21.134 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:316: auto_repair_erasure_coded: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2019-12-21T16:03:21.135 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:319: auto_repair_erasure_coded: teardown td/osd-scrub-repair 2019-12-21T16:03:21.136 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2019-12-21T16:03:21.136 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2019-12-21T16:03:21.136 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2019-12-21T16:03:21.136 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2019-12-21T16:03:21.136 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2019-12-21T16:03:21.136 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2019-12-21T16:03:21.137 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2019-12-21T16:03:21.137 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2019-12-21T16:03:21.243 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2019-12-21T16:03:21.244 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2019-12-21T16:03:21.244 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2019-12-21T16:03:21.245 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2019-12-21T16:03:21.245 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2019-12-21T16:03:21.246 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2019-12-21T16:03:21.246 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2019-12-21T16:03:21.249 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local 'pattern=|/usr/lib/systemd/systemd-coredump %P %u %g %s %t %c %h %e' 2019-12-21T16:03:21.249 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' '|' = '|' ']' 2019-12-21T16:03:21.249 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:177: teardown: pattern= 2019-12-21T16:03:21.249 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2019-12-21T16:03:21.249 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname '' 2019-12-21T16:03:21.250 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls . 2019-12-21T16:03:21.250 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2019-12-21T16:03:21.250 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2019-12-21T16:03:21.261 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2019-12-21T16:03:21.261 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T16:03:21.261 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.119575 2019-12-21T16:03:21.261 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.119575 2019-12-21T16:03:21.263 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2019-12-21T16:03:21.263 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2019-12-21T16:03:21.264 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:62: run: for func in $funcs 2019-12-21T16:03:21.264 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: TEST_auto_repair_erasure_coded_overwrites td/osd-scrub-repair 2019-12-21T16:03:21.264 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:327: TEST_auto_repair_erasure_coded_overwrites: '[' true = true ']' 2019-12-21T16:03:21.264 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:328: TEST_auto_repair_erasure_coded_overwrites: auto_repair_erasure_coded td/osd-scrub-repair true 2019-12-21T16:03:21.264 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:274: auto_repair_erasure_coded: local dir=td/osd-scrub-repair 2019-12-21T16:03:21.264 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:275: auto_repair_erasure_coded: local allow_overwrites=true 2019-12-21T16:03:21.265 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:276: auto_repair_erasure_coded: local poolname=ecpool 2019-12-21T16:03:21.265 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:279: auto_repair_erasure_coded: setup td/osd-scrub-repair 2019-12-21T16:03:21.265 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2019-12-21T16:03:21.265 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2019-12-21T16:03:21.265 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2019-12-21T16:03:21.265 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2019-12-21T16:03:21.266 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2019-12-21T16:03:21.266 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2019-12-21T16:03:21.266 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2019-12-21T16:03:21.266 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2019-12-21T16:03:21.266 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2019-12-21T16:03:21.266 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2019-12-21T16:03:21.267 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2019-12-21T16:03:21.267 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2019-12-21T16:03:21.268 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2019-12-21T16:03:21.268 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2019-12-21T16:03:21.268 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2019-12-21T16:03:21.268 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2019-12-21T16:03:21.268 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2019-12-21T16:03:21.269 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local 'pattern=|/usr/lib/systemd/systemd-coredump %P %u %g %s %t %c %h %e' 2019-12-21T16:03:21.269 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' '|' = '|' ']' 2019-12-21T16:03:21.269 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:177: teardown: pattern= 2019-12-21T16:03:21.269 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2019-12-21T16:03:21.269 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname '' 2019-12-21T16:03:21.270 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls . 2019-12-21T16:03:21.270 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2019-12-21T16:03:21.270 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2019-12-21T16:03:21.271 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2019-12-21T16:03:21.271 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T16:03:21.271 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.119575 2019-12-21T16:03:21.272 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.119575 2019-12-21T16:03:21.272 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2019-12-21T16:03:21.272 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2019-12-21T16:03:21.272 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2019-12-21T16:03:21.273 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2019-12-21T16:03:21.273 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T16:03:21.273 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.119575 2019-12-21T16:03:21.274 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.119575 2019-12-21T16:03:21.275 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2019-12-21T16:03:21.275 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2019-12-21T16:03:21.275 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2019-12-21T16:03:21.276 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2019-12-21T16:03:21.276 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:280: auto_repair_erasure_coded: run_mon td/osd-scrub-repair a 2019-12-21T16:03:21.276 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2019-12-21T16:03:21.276 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2019-12-21T16:03:21.276 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2019-12-21T16:03:21.276 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2019-12-21T16:03:21.277 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2019-12-21T16:03:21.277 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2019-12-21T16:03:21.571 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2019-12-21T16:03:21.572 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2019-12-21T16:03:21.572 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2019-12-21T16:03:21.572 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2019-12-21T16:03:21.572 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T16:03:21.572 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.119575 2019-12-21T16:03:21.572 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.119575/$cluster-$name.asok' 2019-12-21T16:03:21.573 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.119575/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 2019-12-21T16:03:21.747 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: cat 2019-12-21T16:03:21.749 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: get_config mon a fsid 2019-12-21T16:03:21.749 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: get_config: local daemon=mon 2019-12-21T16:03:21.749 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: get_config: local id=a 2019-12-21T16:03:21.749 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local config=fsid 2019-12-21T16:03:21.749 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: jq -r .fsid 2019-12-21T16:03:21.750 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: get_asok_path mon.a 2019-12-21T16:03:21.750 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2019-12-21T16:03:21.750 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2019-12-21T16:03:21.750 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T16:03:21.750 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T16:03:21.750 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.119575 2019-12-21T16:03:21.750 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.119575/ceph-mon.a.asok 2019-12-21T16:03:21.751 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: CEPH_ARGS= 2019-12-21T16:03:21.751 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: ceph --format json daemon /tmp/ceph-asok.119575/ceph-mon.a.asok config get fsid 2019-12-21T16:03:21.836 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: get_config mon a mon_host 2019-12-21T16:03:21.837 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: get_config: local daemon=mon 2019-12-21T16:03:21.837 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: get_config: local id=a 2019-12-21T16:03:21.837 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local config=mon_host 2019-12-21T16:03:21.837 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: jq -r .mon_host 2019-12-21T16:03:21.837 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: get_asok_path mon.a 2019-12-21T16:03:21.837 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2019-12-21T16:03:21.838 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2019-12-21T16:03:21.838 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T16:03:21.838 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T16:03:21.838 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.119575 2019-12-21T16:03:21.838 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.119575/ceph-mon.a.asok 2019-12-21T16:03:21.838 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: CEPH_ARGS= 2019-12-21T16:03:21.838 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: ceph --format json daemon /tmp/ceph-asok.119575/ceph-mon.a.asok config get mon_host 2019-12-21T16:03:21.924 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:281: auto_repair_erasure_coded: run_mgr td/osd-scrub-repair x 2019-12-21T16:03:21.924 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:551: run_mgr: local dir=td/osd-scrub-repair 2019-12-21T16:03:21.925 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: shift 2019-12-21T16:03:21.925 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local id=x 2019-12-21T16:03:21.925 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2019-12-21T16:03:21.925 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local data=td/osd-scrub-repair/x 2019-12-21T16:03:21.925 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:558: run_mgr: get_asok_path 2019-12-21T16:03:21.925 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2019-12-21T16:03:21.925 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2019-12-21T16:03:21.926 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2019-12-21T16:03:21.926 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T16:03:21.926 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.119575 2019-12-21T16:03:21.926 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.119575/$cluster-$name.asok' 2019-12-21T16:03:21.927 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:558: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2019-12-21T16:03:21.927 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:558: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.119575/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2019-12-21T16:03:22.547 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:286: auto_repair_erasure_coded: local 'ceph_osd_args=--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2019-12-21T16:03:22.547 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:287: auto_repair_erasure_coded: seq 0 2 2019-12-21T16:03:22.551 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:287: auto_repair_erasure_coded: for id in $(seq 0 2) 2019-12-21T16:03:22.551 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:288: auto_repair_erasure_coded: '[' true = true ']' 2019-12-21T16:03:22.551 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:289: auto_repair_erasure_coded: run_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2019-12-21T16:03:22.552 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local dir=td/osd-scrub-repair 2019-12-21T16:03:22.552 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: shift 2019-12-21T16:03:22.552 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local id=0 2019-12-21T16:03:22.552 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2019-12-21T16:03:22.552 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local osd_data=td/osd-scrub-repair/0 2019-12-21T16:03:22.552 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local 'ceph_args=--fsid=973e3fc1-e63e-4bde-9db6-84c68fd5e93e --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2019-12-21T16:03:22.552 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2019-12-21T16:03:22.553 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-journal-size=100' 2019-12-21T16:03:22.553 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2019-12-21T16:03:22.553 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2019-12-21T16:03:22.553 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2019-12-21T16:03:22.553 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --chdir=' 2019-12-21T16:03:22.553 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+= 2019-12-21T16:03:22.553 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2019-12-21T16:03:22.554 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: get_asok_path 2019-12-21T16:03:22.554 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2019-12-21T16:03:22.554 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2019-12-21T16:03:22.554 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2019-12-21T16:03:22.554 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T16:03:22.554 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.119575 2019-12-21T16:03:22.555 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.119575/$cluster-$name.asok' 2019-12-21T16:03:22.555 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.119575/$cluster-$name.asok' 2019-12-21T16:03:22.555 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --debug-osd=20' 2019-12-21T16:03:22.555 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-ms=1' 2019-12-21T16:03:22.555 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-monc=20' 2019-12-21T16:03:22.555 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2019-12-21T16:03:22.555 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2019-12-21T16:03:22.555 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2019-12-21T16:03:22.556 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2019-12-21T16:03:22.556 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2019-12-21T16:03:22.556 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' ' 2019-12-21T16:03:22.556 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2019-12-21T16:03:22.556 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: mkdir -p td/osd-scrub-repair/0 2019-12-21T16:03:22.557 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: uuidgen 2019-12-21T16:03:22.557 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: local uuid=e24f0670-6a6b-4445-aa2d-5012b9420715 2019-12-21T16:03:22.557 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: echo 'add osd0 e24f0670-6a6b-4445-aa2d-5012b9420715' 2019-12-21T16:03:22.558 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: ceph-authtool --gen-print-key 2019-12-21T16:03:22.558 INFO:tasks.workunit.client.0.smithi172.stdout:add osd0 e24f0670-6a6b-4445-aa2d-5012b9420715 2019-12-21T16:03:22.565 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: OSD_SECRET=AQBKQv5d3hmXIRAAwpokf8ZNV9alfwWNTNrcIQ== 2019-12-21T16:03:22.565 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo '{"cephx_secret": "AQBKQv5d3hmXIRAAwpokf8ZNV9alfwWNTNrcIQ=="}' 2019-12-21T16:03:22.565 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph osd new e24f0670-6a6b-4445-aa2d-5012b9420715 -i td/osd-scrub-repair/0/new.json 2019-12-21T16:03:22.814 INFO:tasks.workunit.client.0.smithi172.stdout:0 2019-12-21T16:03:22.823 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: rm td/osd-scrub-repair/0/new.json 2019-12-21T16:03:22.824 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-osd -i 0 --fsid=973e3fc1-e63e-4bde-9db6-84c68fd5e93e --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.119575/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQBKQv5d3hmXIRAAwpokf8ZNV9alfwWNTNrcIQ== --osd-uuid e24f0670-6a6b-4445-aa2d-5012b9420715 2019-12-21T16:03:22.848 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:03:22.846+0000 7f69eb116ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T16:03:22.850 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:03:22.847+0000 7f69eb116ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T16:03:22.852 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:03:22.849+0000 7f69eb116ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T16:03:22.853 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:03:22.850+0000 7f69eb116ec0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2019-12-21T16:03:22.853 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:03:22.850+0000 7f69eb116ec0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2019-12-21T16:03:22.853 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:03:22.850+0000 7f69eb116ec0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2019-12-21T16:03:22.897 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:03:22.894+0000 7f69eb116ec0 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2019-12-21T16:03:25.924 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2019-12-21T16:03:25.924 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: cat 2019-12-21T16:03:25.925 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: echo adding osd0 key to auth repository 2019-12-21T16:03:25.925 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2019-12-21T16:03:25.925 INFO:tasks.workunit.client.0.smithi172.stdout:adding osd0 key to auth repository 2019-12-21T16:03:26.117 INFO:tasks.workunit.client.0.smithi172.stdout:start osd.0 2019-12-21T16:03:26.117 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo start osd.0 2019-12-21T16:03:26.117 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph-osd -i 0 --fsid=973e3fc1-e63e-4bde-9db6-84c68fd5e93e --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.119575/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2019-12-21T16:03:26.117 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph osd dump --format=json 2019-12-21T16:03:26.117 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: jq '.flags_set[]' 2019-12-21T16:03:26.118 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: grep -q '"noup"' 2019-12-21T16:03:26.139 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:03:26.136+0000 7f38d8bbcec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T16:03:26.140 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:03:26.138+0000 7f38d8bbcec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T16:03:26.143 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:03:26.140+0000 7f38d8bbcec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T16:03:26.345 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: wait_for_osd up 0 2019-12-21T16:03:26.345 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:914: wait_for_osd: local state=up 2019-12-21T16:03:26.345 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local id=0 2019-12-21T16:03:26.346 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: status=1 2019-12-21T16:03:26.346 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i=0 )) 2019-12-21T16:03:26.346 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i < 300 )) 2019-12-21T16:03:26.346 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: echo 0 2019-12-21T16:03:26.346 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: ceph osd dump 2019-12-21T16:03:26.346 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: grep 'osd.0 up' 2019-12-21T16:03:26.347 INFO:tasks.workunit.client.0.smithi172.stdout:0 2019-12-21T16:03:26.572 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: sleep 1 2019-12-21T16:03:26.659 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:03:26.657+0000 7f38d8bbcec0 -1 Falling back to public interface 2019-12-21T16:03:27.396 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:03:27.395+0000 7f38d8bbcec0 -1 osd.0 0 log_to_monitors {default=true} 2019-12-21T16:03:27.574 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i++ )) 2019-12-21T16:03:27.574 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i < 300 )) 2019-12-21T16:03:27.574 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: echo 1 2019-12-21T16:03:27.574 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: ceph osd dump 2019-12-21T16:03:27.574 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: grep 'osd.0 up' 2019-12-21T16:03:27.575 INFO:tasks.workunit.client.0.smithi172.stdout:1 2019-12-21T16:03:27.868 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: sleep 1 2019-12-21T16:03:28.870 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i++ )) 2019-12-21T16:03:28.870 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i < 300 )) 2019-12-21T16:03:28.870 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: echo 2 2019-12-21T16:03:28.870 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: ceph osd dump 2019-12-21T16:03:28.871 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: grep 'osd.0 up' 2019-12-21T16:03:28.871 INFO:tasks.workunit.client.0.smithi172.stdout:2 2019-12-21T16:03:29.102 INFO:tasks.workunit.client.0.smithi172.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/149057,v1:127.0.0.1:6803/149057] [v2:127.0.0.1:6804/149057,v1:127.0.0.1:6805/149057] exists,up e24f0670-6a6b-4445-aa2d-5012b9420715 2019-12-21T16:03:29.103 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=0 2019-12-21T16:03:29.103 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: break 2019-12-21T16:03:29.103 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: return 0 2019-12-21T16:03:29.103 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:287: auto_repair_erasure_coded: for id in $(seq 0 2) 2019-12-21T16:03:29.103 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:288: auto_repair_erasure_coded: '[' true = true ']' 2019-12-21T16:03:29.103 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:289: auto_repair_erasure_coded: run_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2019-12-21T16:03:29.104 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local dir=td/osd-scrub-repair 2019-12-21T16:03:29.104 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: shift 2019-12-21T16:03:29.104 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local id=1 2019-12-21T16:03:29.104 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2019-12-21T16:03:29.104 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local osd_data=td/osd-scrub-repair/1 2019-12-21T16:03:29.104 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local 'ceph_args=--fsid=973e3fc1-e63e-4bde-9db6-84c68fd5e93e --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2019-12-21T16:03:29.104 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2019-12-21T16:03:29.105 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-journal-size=100' 2019-12-21T16:03:29.105 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2019-12-21T16:03:29.105 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2019-12-21T16:03:29.105 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2019-12-21T16:03:29.105 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --chdir=' 2019-12-21T16:03:29.105 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+= 2019-12-21T16:03:29.105 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2019-12-21T16:03:29.105 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: get_asok_path 2019-12-21T16:03:29.106 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2019-12-21T16:03:29.106 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2019-12-21T16:03:29.106 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2019-12-21T16:03:29.106 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T16:03:29.106 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.119575 2019-12-21T16:03:29.106 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.119575/$cluster-$name.asok' 2019-12-21T16:03:29.106 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.119575/$cluster-$name.asok' 2019-12-21T16:03:29.107 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --debug-osd=20' 2019-12-21T16:03:29.107 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-ms=1' 2019-12-21T16:03:29.107 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-monc=20' 2019-12-21T16:03:29.107 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2019-12-21T16:03:29.107 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2019-12-21T16:03:29.107 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2019-12-21T16:03:29.107 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2019-12-21T16:03:29.108 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2019-12-21T16:03:29.108 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' ' 2019-12-21T16:03:29.108 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2019-12-21T16:03:29.108 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: mkdir -p td/osd-scrub-repair/1 2019-12-21T16:03:29.108 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: uuidgen 2019-12-21T16:03:29.109 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: local uuid=26b5b539-7134-484d-9f48-beaa93e824d7 2019-12-21T16:03:29.109 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: echo 'add osd1 26b5b539-7134-484d-9f48-beaa93e824d7' 2019-12-21T16:03:29.109 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: ceph-authtool --gen-print-key 2019-12-21T16:03:29.109 INFO:tasks.workunit.client.0.smithi172.stdout:add osd1 26b5b539-7134-484d-9f48-beaa93e824d7 2019-12-21T16:03:29.115 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: OSD_SECRET=AQBRQv5dG9DHBhAAUvpmHl7t6wKegTgzQxPXpQ== 2019-12-21T16:03:29.115 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo '{"cephx_secret": "AQBRQv5dG9DHBhAAUvpmHl7t6wKegTgzQxPXpQ=="}' 2019-12-21T16:03:29.115 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph osd new 26b5b539-7134-484d-9f48-beaa93e824d7 -i td/osd-scrub-repair/1/new.json 2019-12-21T16:03:29.445 INFO:tasks.workunit.client.0.smithi172.stdout:1 2019-12-21T16:03:29.455 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: rm td/osd-scrub-repair/1/new.json 2019-12-21T16:03:29.456 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-osd -i 1 --fsid=973e3fc1-e63e-4bde-9db6-84c68fd5e93e --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.119575/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQBRQv5dG9DHBhAAUvpmHl7t6wKegTgzQxPXpQ== --osd-uuid 26b5b539-7134-484d-9f48-beaa93e824d7 2019-12-21T16:03:29.480 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:03:29.478+0000 7fd4444deec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T16:03:29.482 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:03:29.480+0000 7fd4444deec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T16:03:29.484 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:03:29.482+0000 7fd4444deec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T16:03:29.484 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:03:29.483+0000 7fd4444deec0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2019-12-21T16:03:29.485 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:03:29.483+0000 7fd4444deec0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2019-12-21T16:03:29.485 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:03:29.483+0000 7fd4444deec0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2019-12-21T16:03:29.545 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:03:29.543+0000 7fd4444deec0 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2019-12-21T16:03:32.649 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2019-12-21T16:03:32.649 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: cat 2019-12-21T16:03:32.650 INFO:tasks.workunit.client.0.smithi172.stdout:adding osd1 key to auth repository 2019-12-21T16:03:32.650 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: echo adding osd1 key to auth repository 2019-12-21T16:03:32.651 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2019-12-21T16:03:32.934 INFO:tasks.workunit.client.0.smithi172.stdout:start osd.1 2019-12-21T16:03:32.935 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo start osd.1 2019-12-21T16:03:32.935 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph-osd -i 1 --fsid=973e3fc1-e63e-4bde-9db6-84c68fd5e93e --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.119575/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2019-12-21T16:03:32.935 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph osd dump --format=json 2019-12-21T16:03:32.935 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: jq '.flags_set[]' 2019-12-21T16:03:32.935 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: grep -q '"noup"' 2019-12-21T16:03:32.957 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:03:32.955+0000 7fbfeba94ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T16:03:32.958 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:03:32.957+0000 7fbfeba94ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T16:03:32.959 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:03:32.958+0000 7fbfeba94ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T16:03:33.165 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: wait_for_osd up 1 2019-12-21T16:03:33.165 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:914: wait_for_osd: local state=up 2019-12-21T16:03:33.166 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local id=1 2019-12-21T16:03:33.166 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: status=1 2019-12-21T16:03:33.166 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i=0 )) 2019-12-21T16:03:33.166 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i < 300 )) 2019-12-21T16:03:33.166 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: echo 0 2019-12-21T16:03:33.167 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: ceph osd dump 2019-12-21T16:03:33.167 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: grep 'osd.1 up' 2019-12-21T16:03:33.167 INFO:tasks.workunit.client.0.smithi172.stdout:0 2019-12-21T16:03:33.392 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: sleep 1 2019-12-21T16:03:33.476 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:03:33.474+0000 7fbfeba94ec0 -1 Falling back to public interface 2019-12-21T16:03:34.210 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:03:34.209+0000 7fbfeba94ec0 -1 osd.1 0 log_to_monitors {default=true} 2019-12-21T16:03:34.394 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i++ )) 2019-12-21T16:03:34.394 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i < 300 )) 2019-12-21T16:03:34.394 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: echo 1 2019-12-21T16:03:34.395 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: ceph osd dump 2019-12-21T16:03:34.395 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: grep 'osd.1 up' 2019-12-21T16:03:34.395 INFO:tasks.workunit.client.0.smithi172.stdout:1 2019-12-21T16:03:34.631 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: sleep 1 2019-12-21T16:03:35.632 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i++ )) 2019-12-21T16:03:35.632 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i < 300 )) 2019-12-21T16:03:35.632 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: echo 2 2019-12-21T16:03:35.633 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: ceph osd dump 2019-12-21T16:03:35.633 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: grep 'osd.1 up' 2019-12-21T16:03:35.633 INFO:tasks.workunit.client.0.smithi172.stdout:2 2019-12-21T16:03:35.871 INFO:tasks.workunit.client.0.smithi172.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/149434,v1:127.0.0.1:6811/149434] [v2:127.0.0.1:6812/149434,v1:127.0.0.1:6813/149434] exists,up 26b5b539-7134-484d-9f48-beaa93e824d7 2019-12-21T16:03:35.871 INFO:tasks.workunit.client.0.smithi172.stdout:add osd2 8c5abde8-bdd9-40c1-998a-d5f7cfb96d75 2019-12-21T16:03:35.871 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=0 2019-12-21T16:03:35.871 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: break 2019-12-21T16:03:35.871 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: return 0 2019-12-21T16:03:35.872 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:287: auto_repair_erasure_coded: for id in $(seq 0 2) 2019-12-21T16:03:35.872 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:288: auto_repair_erasure_coded: '[' true = true ']' 2019-12-21T16:03:35.872 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:289: auto_repair_erasure_coded: run_osd td/osd-scrub-repair 2 --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2019-12-21T16:03:35.872 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local dir=td/osd-scrub-repair 2019-12-21T16:03:35.872 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: shift 2019-12-21T16:03:35.872 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local id=2 2019-12-21T16:03:35.873 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2019-12-21T16:03:35.873 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local osd_data=td/osd-scrub-repair/2 2019-12-21T16:03:35.873 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local 'ceph_args=--fsid=973e3fc1-e63e-4bde-9db6-84c68fd5e93e --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2019-12-21T16:03:35.873 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2019-12-21T16:03:35.873 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-journal-size=100' 2019-12-21T16:03:35.873 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2019-12-21T16:03:35.873 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2019-12-21T16:03:35.874 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2019-12-21T16:03:35.874 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --chdir=' 2019-12-21T16:03:35.874 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+= 2019-12-21T16:03:35.874 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2019-12-21T16:03:35.874 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: get_asok_path 2019-12-21T16:03:35.874 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2019-12-21T16:03:35.874 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2019-12-21T16:03:35.875 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2019-12-21T16:03:35.875 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T16:03:35.875 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.119575 2019-12-21T16:03:35.875 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.119575/$cluster-$name.asok' 2019-12-21T16:03:35.875 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.119575/$cluster-$name.asok' 2019-12-21T16:03:35.875 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --debug-osd=20' 2019-12-21T16:03:35.875 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-ms=1' 2019-12-21T16:03:35.876 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-monc=20' 2019-12-21T16:03:35.876 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2019-12-21T16:03:35.876 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2019-12-21T16:03:35.876 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2019-12-21T16:03:35.876 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2019-12-21T16:03:35.876 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2019-12-21T16:03:35.877 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' ' 2019-12-21T16:03:35.877 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2019-12-21T16:03:35.877 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: mkdir -p td/osd-scrub-repair/2 2019-12-21T16:03:35.877 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: uuidgen 2019-12-21T16:03:35.877 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: local uuid=8c5abde8-bdd9-40c1-998a-d5f7cfb96d75 2019-12-21T16:03:35.877 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: echo 'add osd2 8c5abde8-bdd9-40c1-998a-d5f7cfb96d75' 2019-12-21T16:03:35.877 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: ceph-authtool --gen-print-key 2019-12-21T16:03:35.883 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: OSD_SECRET=AQBXQv5dHnyPNBAAy9dTwso1OvxwJh96IGXoPw== 2019-12-21T16:03:35.883 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo '{"cephx_secret": "AQBXQv5dHnyPNBAAy9dTwso1OvxwJh96IGXoPw=="}' 2019-12-21T16:03:35.884 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph osd new 8c5abde8-bdd9-40c1-998a-d5f7cfb96d75 -i td/osd-scrub-repair/2/new.json 2019-12-21T16:03:36.218 INFO:tasks.workunit.client.0.smithi172.stdout:2 2019-12-21T16:03:36.228 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: rm td/osd-scrub-repair/2/new.json 2019-12-21T16:03:36.229 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-osd -i 2 --fsid=973e3fc1-e63e-4bde-9db6-84c68fd5e93e --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.119575/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQBXQv5dHnyPNBAAy9dTwso1OvxwJh96IGXoPw== --osd-uuid 8c5abde8-bdd9-40c1-998a-d5f7cfb96d75 2019-12-21T16:03:36.253 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:03:36.251+0000 7f76b448dec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T16:03:36.255 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:03:36.253+0000 7f76b448dec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T16:03:36.256 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:03:36.254+0000 7f76b448dec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T16:03:36.257 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:03:36.255+0000 7f76b448dec0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2019-12-21T16:03:36.257 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:03:36.255+0000 7f76b448dec0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2019-12-21T16:03:36.257 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:03:36.255+0000 7f76b448dec0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2019-12-21T16:03:36.309 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:03:36.308+0000 7f76b448dec0 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2019-12-21T16:03:39.252 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2019-12-21T16:03:39.253 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: cat 2019-12-21T16:03:39.256 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: echo adding osd2 key to auth repository 2019-12-21T16:03:39.256 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2019-12-21T16:03:39.256 INFO:tasks.workunit.client.0.smithi172.stdout:adding osd2 key to auth repository 2019-12-21T16:03:39.538 INFO:tasks.workunit.client.0.smithi172.stdout:start osd.2 2019-12-21T16:03:39.539 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo start osd.2 2019-12-21T16:03:39.539 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph-osd -i 2 --fsid=973e3fc1-e63e-4bde-9db6-84c68fd5e93e --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.119575/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2019-12-21T16:03:39.539 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph osd dump --format=json 2019-12-21T16:03:39.539 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: jq '.flags_set[]' 2019-12-21T16:03:39.540 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: grep -q '"noup"' 2019-12-21T16:03:39.560 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:03:39.558+0000 7fe7dbc25ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T16:03:39.562 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:03:39.560+0000 7fe7dbc25ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T16:03:39.564 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:03:39.562+0000 7fe7dbc25ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T16:03:39.775 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: wait_for_osd up 2 2019-12-21T16:03:39.775 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:914: wait_for_osd: local state=up 2019-12-21T16:03:39.775 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local id=2 2019-12-21T16:03:39.775 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: status=1 2019-12-21T16:03:39.775 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i=0 )) 2019-12-21T16:03:39.776 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i < 300 )) 2019-12-21T16:03:39.776 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: echo 0 2019-12-21T16:03:39.776 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: ceph osd dump 2019-12-21T16:03:39.776 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: grep 'osd.2 up' 2019-12-21T16:03:39.776 INFO:tasks.workunit.client.0.smithi172.stdout:0 2019-12-21T16:03:40.012 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: sleep 1 2019-12-21T16:03:40.082 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:03:40.080+0000 7fe7dbc25ec0 -1 Falling back to public interface 2019-12-21T16:03:40.818 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:03:40.817+0000 7fe7dbc25ec0 -1 osd.2 0 log_to_monitors {default=true} 2019-12-21T16:03:41.013 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i++ )) 2019-12-21T16:03:41.013 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i < 300 )) 2019-12-21T16:03:41.013 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: echo 1 2019-12-21T16:03:41.013 INFO:tasks.workunit.client.0.smithi172.stdout:1 2019-12-21T16:03:41.014 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: ceph osd dump 2019-12-21T16:03:41.014 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: grep 'osd.2 up' 2019-12-21T16:03:41.250 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: sleep 1 2019-12-21T16:03:42.252 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i++ )) 2019-12-21T16:03:42.252 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i < 300 )) 2019-12-21T16:03:42.252 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: echo 2 2019-12-21T16:03:42.253 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: ceph osd dump 2019-12-21T16:03:42.253 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: grep 'osd.2 up' 2019-12-21T16:03:42.253 INFO:tasks.workunit.client.0.smithi172.stdout:2 2019-12-21T16:03:42.489 INFO:tasks.workunit.client.0.smithi172.stdout:osd.2 up in weight 1 up_from 17 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/149811,v1:127.0.0.1:6819/149811] [v2:127.0.0.1:6820/149811,v1:127.0.0.1:6821/149811] exists,up 8c5abde8-bdd9-40c1-998a-d5f7cfb96d75 2019-12-21T16:03:42.489 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=0 2019-12-21T16:03:42.489 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: break 2019-12-21T16:03:42.490 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: return 0 2019-12-21T16:03:42.490 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:294: auto_repair_erasure_coded: create_rbd_pool 2019-12-21T16:03:42.490 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2019-12-21T16:03:42.713 INFO:tasks.workunit.client.0.smithi172.stderr:pool 'rbd' does not exist 2019-12-21T16:03:42.719 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:534: create_rbd_pool: create_pool rbd 4 2019-12-21T16:03:42.719 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:539: create_pool: ceph osd pool create rbd 4 2019-12-21T16:03:43.070 INFO:tasks.workunit.client.0.smithi172.stderr:pool 'rbd' created 2019-12-21T16:03:43.078 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: sleep 1 2019-12-21T16:03:44.079 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: rbd pool init rbd 2019-12-21T16:03:44.720 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:295: auto_repair_erasure_coded: wait_for_clean 2019-12-21T16:03:44.721 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1548: wait_for_clean: local cmd= 2019-12-21T16:03:44.721 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1549: wait_for_clean: local num_active_clean=-1 2019-12-21T16:03:44.721 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1550: wait_for_clean: local cur_active_clean 2019-12-21T16:03:44.721 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1551: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2019-12-21T16:03:44.721 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1551: wait_for_clean: get_timeout_delays 90 .1 2019-12-21T16:03:44.722 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1494: get_timeout_delays: shopt -q -o xtrace 2019-12-21T16:03:44.722 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1494: get_timeout_delays: echo true 2019-12-21T16:03:44.722 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1494: get_timeout_delays: local trace=true 2019-12-21T16:03:44.723 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1495: get_timeout_delays: true 2019-12-21T16:03:44.723 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1495: get_timeout_delays: shopt -u -o xtrace 2019-12-21T16:03:44.816 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1551: wait_for_clean: local -a delays 2019-12-21T16:03:44.816 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1552: wait_for_clean: local -i loop=0 2019-12-21T16:03:44.816 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1554: wait_for_clean: flush_pg_stats 2019-12-21T16:03:44.816 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1977: flush_pg_stats: local timeout=300 2019-12-21T16:03:44.817 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1979: flush_pg_stats: ceph osd ls 2019-12-21T16:03:45.043 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1979: flush_pg_stats: ids='0 2019-12-21T16:03:45.043 INFO:tasks.workunit.client.0.smithi172.stderr:1 2019-12-21T16:03:45.043 INFO:tasks.workunit.client.0.smithi172.stderr:2' 2019-12-21T16:03:45.044 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1980: flush_pg_stats: seqs= 2019-12-21T16:03:45.044 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: flush_pg_stats: for osd in $ids 2019-12-21T16:03:45.044 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2019-12-21T16:03:45.155 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: seq=21474836485 2019-12-21T16:03:45.155 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: flush_pg_stats: seqs=' 0-21474836485' 2019-12-21T16:03:45.155 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: flush_pg_stats: for osd in $ids 2019-12-21T16:03:45.155 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2019-12-21T16:03:45.266 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: seq=42949672964 2019-12-21T16:03:45.266 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: flush_pg_stats: seqs=' 0-21474836485 1-42949672964' 2019-12-21T16:03:45.266 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: flush_pg_stats: for osd in $ids 2019-12-21T16:03:45.266 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2019-12-21T16:03:45.376 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: seq=73014444034 2019-12-21T16:03:45.376 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: flush_pg_stats: seqs=' 0-21474836485 1-42949672964 2-73014444034' 2019-12-21T16:03:45.377 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: flush_pg_stats: for s in $seqs 2019-12-21T16:03:45.377 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: echo 0-21474836485 2019-12-21T16:03:45.377 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: cut -d - -f 1 2019-12-21T16:03:45.378 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: osd=0 2019-12-21T16:03:45.378 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: echo 0-21474836485 2019-12-21T16:03:45.378 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: cut -d - -f 2 2019-12-21T16:03:45.379 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: seq=21474836485 2019-12-21T16:03:45.379 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1989: flush_pg_stats: echo 'waiting osd.0 seq 21474836485' 2019-12-21T16:03:45.380 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 0 2019-12-21T16:03:45.380 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.0 seq 21474836485 2019-12-21T16:03:45.608 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 21474836484 -lt 21474836485 2019-12-21T16:03:45.608 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1991: flush_pg_stats: sleep 1 2019-12-21T16:03:46.609 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1992: flush_pg_stats: '[' 300 -eq 0 ']' 2019-12-21T16:03:46.609 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 0 2019-12-21T16:03:46.839 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 21474836485 -lt 21474836485 2019-12-21T16:03:46.840 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: flush_pg_stats: for s in $seqs 2019-12-21T16:03:46.840 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: echo 1-42949672964 2019-12-21T16:03:46.840 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: cut -d - -f 1 2019-12-21T16:03:46.841 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: osd=1 2019-12-21T16:03:46.842 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: echo 1-42949672964 2019-12-21T16:03:46.842 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: cut -d - -f 2 2019-12-21T16:03:46.842 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: seq=42949672964 2019-12-21T16:03:46.843 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1989: flush_pg_stats: echo 'waiting osd.1 seq 42949672964' 2019-12-21T16:03:46.843 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.1 seq 42949672964 2019-12-21T16:03:46.843 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 1 2019-12-21T16:03:47.071 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 42949672965 -lt 42949672964 2019-12-21T16:03:47.072 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: flush_pg_stats: for s in $seqs 2019-12-21T16:03:47.072 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: echo 2-73014444034 2019-12-21T16:03:47.072 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: cut -d - -f 1 2019-12-21T16:03:47.073 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: osd=2 2019-12-21T16:03:47.073 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: echo 2-73014444034 2019-12-21T16:03:47.073 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: cut -d - -f 2 2019-12-21T16:03:47.074 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: seq=73014444034 2019-12-21T16:03:47.074 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1989: flush_pg_stats: echo 'waiting osd.2 seq 73014444034' 2019-12-21T16:03:47.075 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 2 2019-12-21T16:03:47.075 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.2 seq 73014444034 2019-12-21T16:03:47.322 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 73014444034 -lt 73014444034 2019-12-21T16:03:47.322 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1555: wait_for_clean: get_num_pgs 2019-12-21T16:03:47.323 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: ceph --format json status 2019-12-21T16:03:47.323 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: jq .pgmap.num_pgs 2019-12-21T16:03:47.590 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1555: wait_for_clean: test 4 == 0 2019-12-21T16:03:47.590 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1559: wait_for_clean: true 2019-12-21T16:03:47.590 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1563: wait_for_clean: get_num_active_clean 2019-12-21T16:03:47.590 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: get_num_active_clean: local expression 2019-12-21T16:03:47.591 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2019-12-21T16:03:47.591 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("stale") | not)' 2019-12-21T16:03:47.591 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: ceph --format json pg dump pgs 2019-12-21T16:03:47.591 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2019-12-21T16:03:47.812 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1563: wait_for_clean: cur_active_clean=4 2019-12-21T16:03:47.812 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1564: wait_for_clean: get_num_pgs 2019-12-21T16:03:47.813 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: ceph --format json status 2019-12-21T16:03:47.813 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: jq .pgmap.num_pgs 2019-12-21T16:03:48.083 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1564: wait_for_clean: test 4 = 4 2019-12-21T16:03:48.083 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1564: wait_for_clean: break 2019-12-21T16:03:48.083 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: return 0 2019-12-21T16:03:48.084 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:298: auto_repair_erasure_coded: create_ec_pool ecpool true k=2 m=1 2019-12-21T16:03:48.084 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: create_ec_pool: local pool_name=ecpool 2019-12-21T16:03:48.084 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2179: create_ec_pool: shift 2019-12-21T16:03:48.084 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2180: create_ec_pool: local allow_overwrites=true 2019-12-21T16:03:48.084 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: create_ec_pool: shift 2019-12-21T16:03:48.084 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=1 2019-12-21T16:03:48.456 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2019-12-21T16:03:48.456 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:539: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2019-12-21T16:03:49.183 INFO:tasks.workunit.client.0.smithi172.stderr:pool 'ecpool' created 2019-12-21T16:03:49.191 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: sleep 1 2019-12-21T16:03:50.193 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: create_ec_pool: '[' true = true ']' 2019-12-21T16:03:50.193 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: create_ec_pool: ceph osd pool set ecpool allow_ec_overwrites true 2019-12-21T16:03:50.561 INFO:tasks.workunit.client.0.smithi172.stderr:set pool 2 allow_ec_overwrites to true 2019-12-21T16:03:50.570 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2191: create_ec_pool: wait_for_clean 2019-12-21T16:03:50.570 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1548: wait_for_clean: local cmd= 2019-12-21T16:03:50.570 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1549: wait_for_clean: local num_active_clean=-1 2019-12-21T16:03:50.571 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1550: wait_for_clean: local cur_active_clean 2019-12-21T16:03:50.571 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1551: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2019-12-21T16:03:50.571 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1551: wait_for_clean: get_timeout_delays 90 .1 2019-12-21T16:03:50.571 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1494: get_timeout_delays: shopt -q -o xtrace 2019-12-21T16:03:50.571 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1494: get_timeout_delays: echo true 2019-12-21T16:03:50.571 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1494: get_timeout_delays: local trace=true 2019-12-21T16:03:50.571 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1495: get_timeout_delays: true 2019-12-21T16:03:50.572 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1495: get_timeout_delays: shopt -u -o xtrace 2019-12-21T16:03:50.663 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1551: wait_for_clean: local -a delays 2019-12-21T16:03:50.663 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1552: wait_for_clean: local -i loop=0 2019-12-21T16:03:50.663 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1554: wait_for_clean: flush_pg_stats 2019-12-21T16:03:50.664 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1977: flush_pg_stats: local timeout=300 2019-12-21T16:03:50.664 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1979: flush_pg_stats: ceph osd ls 2019-12-21T16:03:50.891 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1979: flush_pg_stats: ids='0 2019-12-21T16:03:50.891 INFO:tasks.workunit.client.0.smithi172.stderr:1 2019-12-21T16:03:50.891 INFO:tasks.workunit.client.0.smithi172.stderr:2' 2019-12-21T16:03:50.892 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1980: flush_pg_stats: seqs= 2019-12-21T16:03:50.892 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: flush_pg_stats: for osd in $ids 2019-12-21T16:03:50.892 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2019-12-21T16:03:51.003 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: seq=21474836487 2019-12-21T16:03:51.003 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: flush_pg_stats: seqs=' 0-21474836487' 2019-12-21T16:03:51.003 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: flush_pg_stats: for osd in $ids 2019-12-21T16:03:51.003 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2019-12-21T16:03:51.113 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: seq=42949672967 2019-12-21T16:03:51.113 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: flush_pg_stats: seqs=' 0-21474836487 1-42949672967' 2019-12-21T16:03:51.115 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: flush_pg_stats: for osd in $ids 2019-12-21T16:03:51.115 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2019-12-21T16:03:51.224 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: seq=73014444036 2019-12-21T16:03:51.224 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: flush_pg_stats: seqs=' 0-21474836487 1-42949672967 2-73014444036' 2019-12-21T16:03:51.224 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: flush_pg_stats: for s in $seqs 2019-12-21T16:03:51.225 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: echo 0-21474836487 2019-12-21T16:03:51.225 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: cut -d - -f 1 2019-12-21T16:03:51.226 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: osd=0 2019-12-21T16:03:51.226 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: echo 0-21474836487 2019-12-21T16:03:51.226 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: cut -d - -f 2 2019-12-21T16:03:51.227 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: seq=21474836487 2019-12-21T16:03:51.227 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1989: flush_pg_stats: echo 'waiting osd.0 seq 21474836487' 2019-12-21T16:03:51.227 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 0 2019-12-21T16:03:51.227 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.0 seq 21474836487 2019-12-21T16:03:51.482 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 21474836486 -lt 21474836487 2019-12-21T16:03:51.482 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1991: flush_pg_stats: sleep 1 2019-12-21T16:03:52.482 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1992: flush_pg_stats: '[' 300 -eq 0 ']' 2019-12-21T16:03:52.483 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 0 2019-12-21T16:03:52.718 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 21474836487 -lt 21474836487 2019-12-21T16:03:52.718 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: flush_pg_stats: for s in $seqs 2019-12-21T16:03:52.719 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: echo 1-42949672967 2019-12-21T16:03:52.719 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: cut -d - -f 1 2019-12-21T16:03:52.719 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: osd=1 2019-12-21T16:03:52.720 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: echo 1-42949672967 2019-12-21T16:03:52.720 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: cut -d - -f 2 2019-12-21T16:03:52.721 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: seq=42949672967 2019-12-21T16:03:52.721 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1989: flush_pg_stats: echo 'waiting osd.1 seq 42949672967' 2019-12-21T16:03:52.721 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 1 2019-12-21T16:03:52.722 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.1 seq 42949672967 2019-12-21T16:03:52.948 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 42949672967 -lt 42949672967 2019-12-21T16:03:52.949 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: flush_pg_stats: for s in $seqs 2019-12-21T16:03:52.950 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: echo 2-73014444036 2019-12-21T16:03:52.950 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: cut -d - -f 1 2019-12-21T16:03:52.950 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: osd=2 2019-12-21T16:03:52.951 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: echo 2-73014444036 2019-12-21T16:03:52.951 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: cut -d - -f 2 2019-12-21T16:03:52.952 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: seq=73014444036 2019-12-21T16:03:52.955 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1989: flush_pg_stats: echo 'waiting osd.2 seq 73014444036' 2019-12-21T16:03:52.956 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 2 2019-12-21T16:03:52.956 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.2 seq 73014444036 2019-12-21T16:03:53.183 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 73014444036 -lt 73014444036 2019-12-21T16:03:53.184 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1555: wait_for_clean: get_num_pgs 2019-12-21T16:03:53.184 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: ceph --format json status 2019-12-21T16:03:53.184 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: jq .pgmap.num_pgs 2019-12-21T16:03:53.455 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1555: wait_for_clean: test 5 == 0 2019-12-21T16:03:53.456 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1559: wait_for_clean: true 2019-12-21T16:03:53.457 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1563: wait_for_clean: get_num_active_clean 2019-12-21T16:03:53.457 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: get_num_active_clean: local expression 2019-12-21T16:03:53.457 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2019-12-21T16:03:53.457 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("stale") | not)' 2019-12-21T16:03:53.457 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: ceph --format json pg dump pgs 2019-12-21T16:03:53.458 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2019-12-21T16:03:53.681 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1563: wait_for_clean: cur_active_clean=5 2019-12-21T16:03:53.682 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1564: wait_for_clean: get_num_pgs 2019-12-21T16:03:53.682 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: ceph --format json status 2019-12-21T16:03:53.683 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: jq .pgmap.num_pgs 2019-12-21T16:03:53.951 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1564: wait_for_clean: test 5 = 5 2019-12-21T16:03:53.951 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1564: wait_for_clean: break 2019-12-21T16:03:53.952 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: return 0 2019-12-21T16:03:53.952 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2192: create_ec_pool: return 0 2019-12-21T16:03:53.952 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:301: auto_repair_erasure_coded: local payload=ABCDEF 2019-12-21T16:03:53.952 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:302: auto_repair_erasure_coded: echo ABCDEF 2019-12-21T16:03:53.952 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:303: auto_repair_erasure_coded: rados --pool ecpool put SOMETHING td/osd-scrub-repair/ORIGINAL 2019-12-21T16:03:54.111 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:307: auto_repair_erasure_coded: get_not_primary ecpool SOMETHING 2019-12-21T16:03:54.111 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1165: get_not_primary: local poolname=ecpool 2019-12-21T16:03:54.111 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1166: get_not_primary: local objectname=SOMETHING 2019-12-21T16:03:54.111 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: get_not_primary: get_primary ecpool SOMETHING 2019-12-21T16:03:54.111 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_primary: local poolname=ecpool 2019-12-21T16:03:54.112 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1133: get_primary: local objectname=SOMETHING 2019-12-21T16:03:54.112 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1135: get_primary: ceph --format json osd map ecpool SOMETHING 2019-12-21T16:03:54.112 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1136: get_primary: jq .acting_primary 2019-12-21T16:03:54.342 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: get_not_primary: local primary=2 2019-12-21T16:03:54.343 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1169: get_not_primary: ceph --format json osd map ecpool SOMETHING 2019-12-21T16:03:54.343 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1170: get_not_primary: jq '.acting | map(select (. != 2)) | .[0]' 2019-12-21T16:03:54.598 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:307: auto_repair_erasure_coded: objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2019-12-21T16:03:54.599 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: objectstore_tool: local dir=td/osd-scrub-repair 2019-12-21T16:03:54.599 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: objectstore_tool: shift 2019-12-21T16:03:54.599 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1235: objectstore_tool: local id=1 2019-12-21T16:03:54.599 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1236: objectstore_tool: shift 2019-12-21T16:03:54.600 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1238: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING remove 2019-12-21T16:03:54.600 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1205: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2019-12-21T16:03:54.600 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nowait: shift 2019-12-21T16:03:54.600 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1207: _objectstore_tool_nowait: local id=1 2019-12-21T16:03:54.601 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1208: _objectstore_tool_nowait: shift 2019-12-21T16:03:54.601 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1210: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2019-12-21T16:03:54.601 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2019-12-21T16:03:54.601 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2019-12-21T16:03:54.602 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2019-12-21T16:03:54.602 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2019-12-21T16:03:54.602 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2019-12-21T16:03:54.702 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2019-12-21T16:03:54.703 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1212: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING remove 2019-12-21T16:03:54.703 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1193: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2019-12-21T16:03:54.703 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1194: _objectstore_tool_nodown: shift 2019-12-21T16:03:54.704 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1195: _objectstore_tool_nodown: local id=1 2019-12-21T16:03:54.704 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: _objectstore_tool_nodown: shift 2019-12-21T16:03:54.704 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2019-12-21T16:03:54.704 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING remove 2019-12-21T16:03:55.276 INFO:tasks.workunit.client.0.smithi172.stderr:Error getting attr on : 1.3_head,#-3:c0000000:::scrub_1.3:head#, (61) No data available 2019-12-21T16:03:55.276 INFO:tasks.workunit.client.0.smithi172.stderr:Error getting attr on : 1.0_head,#-3:00000000:::scrub_1.0:head#, (61) No data available 2019-12-21T16:03:55.276 INFO:tasks.workunit.client.0.smithi172.stdout:remove 1#2:eb822e21:::SOMETHING:head# 2019-12-21T16:03:55.793 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1213: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2019-12-21T16:03:55.794 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:842: activate_osd: local dir=td/osd-scrub-repair 2019-12-21T16:03:55.794 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: shift 2019-12-21T16:03:55.794 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: local id=1 2019-12-21T16:03:55.794 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: shift 2019-12-21T16:03:55.794 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local osd_data=td/osd-scrub-repair/1 2019-12-21T16:03:55.794 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local 'ceph_args=--fsid=973e3fc1-e63e-4bde-9db6-84c68fd5e93e --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2019-12-21T16:03:55.795 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2019-12-21T16:03:55.795 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-journal-size=100' 2019-12-21T16:03:55.795 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2019-12-21T16:03:55.795 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2019-12-21T16:03:55.795 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2019-12-21T16:03:55.795 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --chdir=' 2019-12-21T16:03:55.795 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+= 2019-12-21T16:03:55.796 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2019-12-21T16:03:55.796 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: get_asok_path 2019-12-21T16:03:55.796 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2019-12-21T16:03:55.796 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2019-12-21T16:03:55.797 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2019-12-21T16:03:55.797 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T16:03:55.798 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.119575 2019-12-21T16:03:55.798 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.119575/$cluster-$name.asok' 2019-12-21T16:03:55.798 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.119575/$cluster-$name.asok' 2019-12-21T16:03:55.798 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --debug-osd=20' 2019-12-21T16:03:55.798 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2019-12-21T16:03:55.798 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2019-12-21T16:03:55.798 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2019-12-21T16:03:55.799 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2019-12-21T16:03:55.799 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2019-12-21T16:03:55.799 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' ' 2019-12-21T16:03:55.799 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2019-12-21T16:03:55.799 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: mkdir -p td/osd-scrub-repair/1 2019-12-21T16:03:55.799 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: echo start osd.1 2019-12-21T16:03:55.799 INFO:tasks.workunit.client.0.smithi172.stderr:start osd.1 2019-12-21T16:03:55.802 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph-osd -i 1 --fsid=973e3fc1-e63e-4bde-9db6-84c68fd5e93e --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.119575/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2019-12-21T16:03:55.802 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: cat td/osd-scrub-repair/1/whoami 2019-12-21T16:03:55.802 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: '[' 1 = 1 ']' 2019-12-21T16:03:55.802 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph osd dump --format=json 2019-12-21T16:03:55.802 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: jq '.flags_set[]' 2019-12-21T16:03:55.802 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: grep -q '"noup"' 2019-12-21T16:03:55.817 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:03:55.815+0000 7f462456dec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T16:03:55.819 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:03:55.817+0000 7f462456dec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T16:03:55.821 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:03:55.818+0000 7f462456dec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T16:03:56.031 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: wait_for_osd up 1 2019-12-21T16:03:56.031 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:914: wait_for_osd: local state=up 2019-12-21T16:03:56.031 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local id=1 2019-12-21T16:03:56.032 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: status=1 2019-12-21T16:03:56.032 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i=0 )) 2019-12-21T16:03:56.032 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i < 300 )) 2019-12-21T16:03:56.032 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: echo 0 2019-12-21T16:03:56.032 INFO:tasks.workunit.client.0.smithi172.stderr:0 2019-12-21T16:03:56.032 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: ceph osd dump 2019-12-21T16:03:56.032 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: grep 'osd.1 up' 2019-12-21T16:03:56.336 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:03:56.334+0000 7f462456dec0 -1 Falling back to public interface 2019-12-21T16:03:56.346 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: sleep 1 2019-12-21T16:03:57.203 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:03:57.200+0000 7f462456dec0 -1 osd.1 38 log_to_monitors {default=true} 2019-12-21T16:03:57.349 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i++ )) 2019-12-21T16:03:57.349 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i < 300 )) 2019-12-21T16:03:57.349 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: echo 1 2019-12-21T16:03:57.349 INFO:tasks.workunit.client.0.smithi172.stderr:1 2019-12-21T16:03:57.349 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: ceph osd dump 2019-12-21T16:03:57.350 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: grep 'osd.1 up' 2019-12-21T16:03:57.661 INFO:tasks.workunit.client.0.smithi172.stderr:osd.1 up in weight 1 up_from 41 up_thru 20 down_at 39 last_clean_interval [10,38) [v2:127.0.0.1:6810/151129,v1:127.0.0.1:6811/151129] [v2:127.0.0.1:6812/151129,v1:127.0.0.1:6813/151129] exists,up 26b5b539-7134-484d-9f48-beaa93e824d7 2019-12-21T16:03:57.661 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=0 2019-12-21T16:03:57.661 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: break 2019-12-21T16:03:57.661 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: return 0 2019-12-21T16:03:57.662 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1239: objectstore_tool: wait_for_clean 2019-12-21T16:03:57.663 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1548: wait_for_clean: local cmd= 2019-12-21T16:03:57.663 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1549: wait_for_clean: local num_active_clean=-1 2019-12-21T16:03:57.663 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1550: wait_for_clean: local cur_active_clean 2019-12-21T16:03:57.664 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1551: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2019-12-21T16:03:57.664 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1551: wait_for_clean: get_timeout_delays 90 .1 2019-12-21T16:03:57.664 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1494: get_timeout_delays: shopt -q -o xtrace 2019-12-21T16:03:57.664 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1494: get_timeout_delays: echo true 2019-12-21T16:03:57.664 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1494: get_timeout_delays: local trace=true 2019-12-21T16:03:57.664 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1495: get_timeout_delays: true 2019-12-21T16:03:57.665 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1495: get_timeout_delays: shopt -u -o xtrace 2019-12-21T16:03:57.753 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1551: wait_for_clean: local -a delays 2019-12-21T16:03:57.753 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1552: wait_for_clean: local -i loop=0 2019-12-21T16:03:57.753 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1554: wait_for_clean: flush_pg_stats 2019-12-21T16:03:57.753 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1977: flush_pg_stats: local timeout=300 2019-12-21T16:03:57.754 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1979: flush_pg_stats: ceph osd ls 2019-12-21T16:03:58.014 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1979: flush_pg_stats: ids='0 2019-12-21T16:03:58.014 INFO:tasks.workunit.client.0.smithi172.stderr:1 2019-12-21T16:03:58.014 INFO:tasks.workunit.client.0.smithi172.stderr:2' 2019-12-21T16:03:58.015 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1980: flush_pg_stats: seqs= 2019-12-21T16:03:58.015 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: flush_pg_stats: for osd in $ids 2019-12-21T16:03:58.015 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2019-12-21T16:03:58.127 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: seq=21474836489 2019-12-21T16:03:58.127 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: flush_pg_stats: seqs=' 0-21474836489' 2019-12-21T16:03:58.128 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: flush_pg_stats: for osd in $ids 2019-12-21T16:03:58.128 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2019-12-21T16:03:58.238 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: seq=176093659138 2019-12-21T16:03:58.238 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: flush_pg_stats: seqs=' 0-21474836489 1-176093659138' 2019-12-21T16:03:58.238 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: flush_pg_stats: for osd in $ids 2019-12-21T16:03:58.239 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2019-12-21T16:03:58.371 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: seq=73014444039 2019-12-21T16:03:58.371 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: flush_pg_stats: seqs=' 0-21474836489 1-176093659138 2-73014444039' 2019-12-21T16:03:58.371 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: flush_pg_stats: for s in $seqs 2019-12-21T16:03:58.372 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: echo 0-21474836489 2019-12-21T16:03:58.372 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: cut -d - -f 1 2019-12-21T16:03:58.372 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: osd=0 2019-12-21T16:03:58.373 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: echo 0-21474836489 2019-12-21T16:03:58.373 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: cut -d - -f 2 2019-12-21T16:03:58.374 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: seq=21474836489 2019-12-21T16:03:58.374 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1989: flush_pg_stats: echo 'waiting osd.0 seq 21474836489' 2019-12-21T16:03:58.374 INFO:tasks.workunit.client.0.smithi172.stderr:waiting osd.0 seq 21474836489 2019-12-21T16:03:58.374 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 0 2019-12-21T16:03:58.601 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 21474836488 -lt 21474836489 2019-12-21T16:03:58.601 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1991: flush_pg_stats: sleep 1 2019-12-21T16:03:59.602 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1992: flush_pg_stats: '[' 300 -eq 0 ']' 2019-12-21T16:03:59.603 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 0 2019-12-21T16:03:59.838 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 21474836488 -lt 21474836489 2019-12-21T16:03:59.838 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1991: flush_pg_stats: sleep 1 2019-12-21T16:04:00.839 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1992: flush_pg_stats: '[' 299 -eq 0 ']' 2019-12-21T16:04:00.840 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 0 2019-12-21T16:04:01.078 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 21474836490 -lt 21474836489 2019-12-21T16:04:01.078 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: flush_pg_stats: for s in $seqs 2019-12-21T16:04:01.079 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: echo 1-176093659138 2019-12-21T16:04:01.079 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: cut -d - -f 1 2019-12-21T16:04:01.079 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: osd=1 2019-12-21T16:04:01.080 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: echo 1-176093659138 2019-12-21T16:04:01.080 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: cut -d - -f 2 2019-12-21T16:04:01.081 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: seq=176093659138 2019-12-21T16:04:01.081 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1989: flush_pg_stats: echo 'waiting osd.1 seq 176093659138' 2019-12-21T16:04:01.081 INFO:tasks.workunit.client.0.smithi172.stderr:waiting osd.1 seq 176093659138 2019-12-21T16:04:01.082 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 1 2019-12-21T16:04:01.326 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 176093659138 -lt 176093659138 2019-12-21T16:04:01.326 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: flush_pg_stats: for s in $seqs 2019-12-21T16:04:01.326 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: echo 2-73014444039 2019-12-21T16:04:01.327 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: cut -d - -f 1 2019-12-21T16:04:01.327 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: osd=2 2019-12-21T16:04:01.330 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: echo 2-73014444039 2019-12-21T16:04:01.331 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: cut -d - -f 2 2019-12-21T16:04:01.331 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: seq=73014444039 2019-12-21T16:04:01.331 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1989: flush_pg_stats: echo 'waiting osd.2 seq 73014444039' 2019-12-21T16:04:01.331 INFO:tasks.workunit.client.0.smithi172.stderr:waiting osd.2 seq 73014444039 2019-12-21T16:04:01.331 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 2 2019-12-21T16:04:01.557 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 73014444039 -lt 73014444039 2019-12-21T16:04:01.557 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1555: wait_for_clean: get_num_pgs 2019-12-21T16:04:01.558 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: ceph --format json status 2019-12-21T16:04:01.558 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: jq .pgmap.num_pgs 2019-12-21T16:04:01.829 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1555: wait_for_clean: test 5 == 0 2019-12-21T16:04:01.829 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1559: wait_for_clean: true 2019-12-21T16:04:01.830 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1563: wait_for_clean: get_num_active_clean 2019-12-21T16:04:01.830 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: get_num_active_clean: local expression 2019-12-21T16:04:01.830 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2019-12-21T16:04:01.830 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("stale") | not)' 2019-12-21T16:04:01.830 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: ceph --format json pg dump pgs 2019-12-21T16:04:01.830 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2019-12-21T16:04:02.053 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1563: wait_for_clean: cur_active_clean=5 2019-12-21T16:04:02.054 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1564: wait_for_clean: get_num_pgs 2019-12-21T16:04:02.054 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: ceph --format json status 2019-12-21T16:04:02.054 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: jq .pgmap.num_pgs 2019-12-21T16:04:02.329 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1564: wait_for_clean: test 5 = 5 2019-12-21T16:04:02.329 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1564: wait_for_clean: break 2019-12-21T16:04:02.329 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: return 0 2019-12-21T16:04:02.329 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:309: auto_repair_erasure_coded: get_pg ecpool SOMETHING 2019-12-21T16:04:02.330 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1029: get_pg: local poolname=ecpool 2019-12-21T16:04:02.330 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1030: get_pg: local objectname=SOMETHING 2019-12-21T16:04:02.330 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1032: get_pg: ceph --format json osd map ecpool SOMETHING 2019-12-21T16:04:02.330 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1032: get_pg: jq -r .pgid 2019-12-21T16:04:02.559 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:309: auto_repair_erasure_coded: local pgid=2.0 2019-12-21T16:04:02.560 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:310: auto_repair_erasure_coded: get_last_scrub_stamp 2.0 2019-12-21T16:04:02.560 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=2.0 2019-12-21T16:04:02.560 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T16:04:02.560 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T16:04:02.561 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2019-12-21T16:04:02.783 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:310: auto_repair_erasure_coded: wait_for_scrub 2.0 2019-12-21T16:03:49.003445+0000 2019-12-21T16:04:02.783 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1789: wait_for_scrub: local pgid=2.0 2019-12-21T16:04:02.784 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1790: wait_for_scrub: local last_scrub=2019-12-21T16:03:49.003445+0000 2019-12-21T16:04:02.784 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1791: wait_for_scrub: local sname=last_scrub_stamp 2019-12-21T16:04:02.784 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i=0 )) 2019-12-21T16:04:02.784 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T16:04:02.784 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2019-12-21T16:04:02.784 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=2.0 2019-12-21T16:04:02.785 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T16:04:02.785 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T16:04:02.785 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2019-12-21T16:04:03.008 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T16:03:49.003445+0000 '>' 2019-12-21T16:03:49.003445+0000 2019-12-21T16:04:03.008 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T16:04:04.008 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T16:04:04.009 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T16:04:04.009 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2019-12-21T16:04:04.010 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=2.0 2019-12-21T16:04:04.010 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T16:04:04.010 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T16:04:04.010 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2019-12-21T16:04:04.247 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T16:03:49.003445+0000 '>' 2019-12-21T16:03:49.003445+0000 2019-12-21T16:04:04.247 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T16:04:05.248 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T16:04:05.248 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T16:04:05.249 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2019-12-21T16:04:05.249 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=2.0 2019-12-21T16:04:05.249 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T16:04:05.249 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T16:04:05.249 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2019-12-21T16:04:05.485 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T16:03:49.003445+0000 '>' 2019-12-21T16:03:49.003445+0000 2019-12-21T16:04:05.485 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T16:04:06.482 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T16:04:06.483 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T16:04:06.483 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2019-12-21T16:04:06.484 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=2.0 2019-12-21T16:04:06.484 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T16:04:06.484 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T16:04:06.484 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2019-12-21T16:04:06.715 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T16:03:49.003445+0000 '>' 2019-12-21T16:03:49.003445+0000 2019-12-21T16:04:06.716 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T16:04:07.716 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T16:04:07.717 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T16:04:07.718 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2019-12-21T16:04:07.718 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=2.0 2019-12-21T16:04:07.719 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T16:04:07.719 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T16:04:07.719 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2019-12-21T16:04:07.947 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T16:04:02.833021+0000 '>' 2019-12-21T16:03:49.003445+0000 2019-12-21T16:04:07.948 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1795: wait_for_scrub: return 0 2019-12-21T16:04:07.948 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:311: auto_repair_erasure_coded: wait_for_clean 2019-12-21T16:04:07.948 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1548: wait_for_clean: local cmd= 2019-12-21T16:04:07.948 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1549: wait_for_clean: local num_active_clean=-1 2019-12-21T16:04:07.948 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1550: wait_for_clean: local cur_active_clean 2019-12-21T16:04:07.948 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1551: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2019-12-21T16:04:07.949 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1551: wait_for_clean: get_timeout_delays 90 .1 2019-12-21T16:04:07.949 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1494: get_timeout_delays: shopt -q -o xtrace 2019-12-21T16:04:07.949 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1494: get_timeout_delays: echo true 2019-12-21T16:04:07.949 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1494: get_timeout_delays: local trace=true 2019-12-21T16:04:07.949 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1495: get_timeout_delays: true 2019-12-21T16:04:07.949 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1495: get_timeout_delays: shopt -u -o xtrace 2019-12-21T16:04:08.040 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1551: wait_for_clean: local -a delays 2019-12-21T16:04:08.040 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1552: wait_for_clean: local -i loop=0 2019-12-21T16:04:08.040 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1554: wait_for_clean: flush_pg_stats 2019-12-21T16:04:08.040 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1977: flush_pg_stats: local timeout=300 2019-12-21T16:04:08.040 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1979: flush_pg_stats: ceph osd ls 2019-12-21T16:04:08.266 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1979: flush_pg_stats: ids='0 2019-12-21T16:04:08.266 INFO:tasks.workunit.client.0.smithi172.stderr:1 2019-12-21T16:04:08.267 INFO:tasks.workunit.client.0.smithi172.stderr:2' 2019-12-21T16:04:08.267 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1980: flush_pg_stats: seqs= 2019-12-21T16:04:08.267 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: flush_pg_stats: for osd in $ids 2019-12-21T16:04:08.267 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2019-12-21T16:04:08.377 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: seq=21474836492 2019-12-21T16:04:08.378 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: flush_pg_stats: seqs=' 0-21474836492' 2019-12-21T16:04:08.378 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: flush_pg_stats: for osd in $ids 2019-12-21T16:04:08.378 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2019-12-21T16:04:08.509 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: seq=176093659141 2019-12-21T16:04:08.509 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: flush_pg_stats: seqs=' 0-21474836492 1-176093659141' 2019-12-21T16:04:08.510 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: flush_pg_stats: for osd in $ids 2019-12-21T16:04:08.510 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2019-12-21T16:04:08.616 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: seq=73014444042 2019-12-21T16:04:08.616 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: flush_pg_stats: seqs=' 0-21474836492 1-176093659141 2-73014444042' 2019-12-21T16:04:08.616 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: flush_pg_stats: for s in $seqs 2019-12-21T16:04:08.616 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: echo 0-21474836492 2019-12-21T16:04:08.616 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: cut -d - -f 1 2019-12-21T16:04:08.617 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: osd=0 2019-12-21T16:04:08.617 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: echo 0-21474836492 2019-12-21T16:04:08.617 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: cut -d - -f 2 2019-12-21T16:04:08.619 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.0 seq 21474836492 2019-12-21T16:04:08.619 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: seq=21474836492 2019-12-21T16:04:08.619 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1989: flush_pg_stats: echo 'waiting osd.0 seq 21474836492' 2019-12-21T16:04:08.620 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 0 2019-12-21T16:04:08.847 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 21474836491 -lt 21474836492 2019-12-21T16:04:08.848 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1991: flush_pg_stats: sleep 1 2019-12-21T16:04:09.847 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1992: flush_pg_stats: '[' 300 -eq 0 ']' 2019-12-21T16:04:09.847 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 0 2019-12-21T16:04:10.085 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 21474836493 -lt 21474836492 2019-12-21T16:04:10.085 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: flush_pg_stats: for s in $seqs 2019-12-21T16:04:10.085 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: echo 1-176093659141 2019-12-21T16:04:10.085 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: cut -d - -f 1 2019-12-21T16:04:10.086 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: osd=1 2019-12-21T16:04:10.086 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: echo 1-176093659141 2019-12-21T16:04:10.087 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: cut -d - -f 2 2019-12-21T16:04:10.087 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: seq=176093659141 2019-12-21T16:04:10.087 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1989: flush_pg_stats: echo 'waiting osd.1 seq 176093659141' 2019-12-21T16:04:10.088 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 1 2019-12-21T16:04:10.088 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.1 seq 176093659141 2019-12-21T16:04:10.316 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 176093659141 -lt 176093659141 2019-12-21T16:04:10.316 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: flush_pg_stats: for s in $seqs 2019-12-21T16:04:10.317 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: echo 2-73014444042 2019-12-21T16:04:10.317 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: cut -d - -f 1 2019-12-21T16:04:10.317 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: osd=2 2019-12-21T16:04:10.318 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: echo 2-73014444042 2019-12-21T16:04:10.318 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: cut -d - -f 2 2019-12-21T16:04:10.319 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: seq=73014444042 2019-12-21T16:04:10.319 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1989: flush_pg_stats: echo 'waiting osd.2 seq 73014444042' 2019-12-21T16:04:10.319 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 2 2019-12-21T16:04:10.319 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.2 seq 73014444042 2019-12-21T16:04:10.561 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 73014444042 -lt 73014444042 2019-12-21T16:04:10.562 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1555: wait_for_clean: get_num_pgs 2019-12-21T16:04:10.562 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: ceph --format json status 2019-12-21T16:04:10.562 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: jq .pgmap.num_pgs 2019-12-21T16:04:10.832 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1555: wait_for_clean: test 5 == 0 2019-12-21T16:04:10.832 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1559: wait_for_clean: true 2019-12-21T16:04:10.832 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1563: wait_for_clean: get_num_active_clean 2019-12-21T16:04:10.833 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: get_num_active_clean: local expression 2019-12-21T16:04:10.833 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2019-12-21T16:04:10.833 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("stale") | not)' 2019-12-21T16:04:10.833 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: ceph --format json pg dump pgs 2019-12-21T16:04:10.833 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2019-12-21T16:04:11.074 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1563: wait_for_clean: cur_active_clean=5 2019-12-21T16:04:11.074 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1564: wait_for_clean: get_num_pgs 2019-12-21T16:04:11.075 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: ceph --format json status 2019-12-21T16:04:11.075 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: jq .pgmap.num_pgs 2019-12-21T16:04:11.343 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1564: wait_for_clean: test 5 = 5 2019-12-21T16:04:11.343 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1564: wait_for_clean: break 2019-12-21T16:04:11.343 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: return 0 2019-12-21T16:04:11.344 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:314: auto_repair_erasure_coded: get_not_primary ecpool SOMETHING 2019-12-21T16:04:11.344 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1165: get_not_primary: local poolname=ecpool 2019-12-21T16:04:11.344 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1166: get_not_primary: local objectname=SOMETHING 2019-12-21T16:04:11.345 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: get_not_primary: get_primary ecpool SOMETHING 2019-12-21T16:04:11.345 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_primary: local poolname=ecpool 2019-12-21T16:04:11.345 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1133: get_primary: local objectname=SOMETHING 2019-12-21T16:04:11.345 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1135: get_primary: ceph --format json osd map ecpool SOMETHING 2019-12-21T16:04:11.345 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1136: get_primary: jq .acting_primary 2019-12-21T16:04:11.574 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: get_not_primary: local primary=2 2019-12-21T16:04:11.574 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1169: get_not_primary: ceph --format json osd map ecpool SOMETHING 2019-12-21T16:04:11.574 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1170: get_not_primary: jq '.acting | map(select (. != 2)) | .[0]' 2019-12-21T16:04:11.809 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:314: auto_repair_erasure_coded: objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2019-12-21T16:04:11.810 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: objectstore_tool: local dir=td/osd-scrub-repair 2019-12-21T16:04:11.810 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: objectstore_tool: shift 2019-12-21T16:04:11.810 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1235: objectstore_tool: local id=1 2019-12-21T16:04:11.810 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1236: objectstore_tool: shift 2019-12-21T16:04:11.810 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1238: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING list-attrs 2019-12-21T16:04:11.810 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1205: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2019-12-21T16:04:11.810 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nowait: shift 2019-12-21T16:04:11.811 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1207: _objectstore_tool_nowait: local id=1 2019-12-21T16:04:11.811 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1208: _objectstore_tool_nowait: shift 2019-12-21T16:04:11.811 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1210: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2019-12-21T16:04:11.811 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2019-12-21T16:04:11.811 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2019-12-21T16:04:11.811 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2019-12-21T16:04:11.811 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2019-12-21T16:04:11.812 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2019-12-21T16:04:11.914 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2019-12-21T16:04:11.915 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1212: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING list-attrs 2019-12-21T16:04:11.915 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1193: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2019-12-21T16:04:11.915 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1194: _objectstore_tool_nodown: shift 2019-12-21T16:04:11.915 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1195: _objectstore_tool_nodown: local id=1 2019-12-21T16:04:11.915 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: _objectstore_tool_nodown: shift 2019-12-21T16:04:11.915 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2019-12-21T16:04:11.915 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING list-attrs 2019-12-21T16:04:12.572 INFO:tasks.workunit.client.0.smithi172.stderr:Error getting attr on : 1.3_head,#-3:c0000000:::scrub_1.3:head#, (61) No data available 2019-12-21T16:04:12.572 INFO:tasks.workunit.client.0.smithi172.stderr:Error getting attr on : 1.0_head,#-3:00000000:::scrub_1.0:head#, (61) No data available 2019-12-21T16:04:12.573 INFO:tasks.workunit.client.0.smithi172.stdout:_ 2019-12-21T16:04:12.573 INFO:tasks.workunit.client.0.smithi172.stdout:hinfo_key 2019-12-21T16:04:12.573 INFO:tasks.workunit.client.0.smithi172.stdout:snapset 2019-12-21T16:04:13.091 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1213: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2019-12-21T16:04:13.091 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:842: activate_osd: local dir=td/osd-scrub-repair 2019-12-21T16:04:13.091 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: shift 2019-12-21T16:04:13.092 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: local id=1 2019-12-21T16:04:13.092 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: shift 2019-12-21T16:04:13.092 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local osd_data=td/osd-scrub-repair/1 2019-12-21T16:04:13.092 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local 'ceph_args=--fsid=973e3fc1-e63e-4bde-9db6-84c68fd5e93e --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2019-12-21T16:04:13.092 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2019-12-21T16:04:13.093 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-journal-size=100' 2019-12-21T16:04:13.093 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2019-12-21T16:04:13.093 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2019-12-21T16:04:13.093 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2019-12-21T16:04:13.093 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --chdir=' 2019-12-21T16:04:13.093 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+= 2019-12-21T16:04:13.093 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2019-12-21T16:04:13.094 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: get_asok_path 2019-12-21T16:04:13.094 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2019-12-21T16:04:13.094 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2019-12-21T16:04:13.094 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2019-12-21T16:04:13.094 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T16:04:13.094 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.119575 2019-12-21T16:04:13.094 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.119575/$cluster-$name.asok' 2019-12-21T16:04:13.095 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.119575/$cluster-$name.asok' 2019-12-21T16:04:13.095 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --debug-osd=20' 2019-12-21T16:04:13.095 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2019-12-21T16:04:13.095 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2019-12-21T16:04:13.095 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2019-12-21T16:04:13.095 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2019-12-21T16:04:13.095 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2019-12-21T16:04:13.096 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' ' 2019-12-21T16:04:13.096 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2019-12-21T16:04:13.096 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: mkdir -p td/osd-scrub-repair/1 2019-12-21T16:04:13.096 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: echo start osd.1 2019-12-21T16:04:13.096 INFO:tasks.workunit.client.0.smithi172.stderr:start osd.1 2019-12-21T16:04:13.096 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph-osd -i 1 --fsid=973e3fc1-e63e-4bde-9db6-84c68fd5e93e --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.119575/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2019-12-21T16:04:13.096 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: cat td/osd-scrub-repair/1/whoami 2019-12-21T16:04:13.097 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: '[' 1 = 1 ']' 2019-12-21T16:04:13.097 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph osd dump --format=json 2019-12-21T16:04:13.098 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: jq '.flags_set[]' 2019-12-21T16:04:13.098 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: grep -q '"noup"' 2019-12-21T16:04:13.114 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:04:13.113+0000 7f87e6ae3ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T16:04:13.116 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:04:13.115+0000 7f87e6ae3ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T16:04:13.119 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:04:13.117+0000 7f87e6ae3ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T16:04:13.329 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: wait_for_osd up 1 2019-12-21T16:04:13.329 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:914: wait_for_osd: local state=up 2019-12-21T16:04:13.329 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local id=1 2019-12-21T16:04:13.329 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: status=1 2019-12-21T16:04:13.329 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i=0 )) 2019-12-21T16:04:13.330 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i < 300 )) 2019-12-21T16:04:13.330 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: echo 0 2019-12-21T16:04:13.330 INFO:tasks.workunit.client.0.smithi172.stderr:0 2019-12-21T16:04:13.330 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: ceph osd dump 2019-12-21T16:04:13.330 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: grep 'osd.1 up' 2019-12-21T16:04:13.557 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: sleep 1 2019-12-21T16:04:13.634 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:04:13.632+0000 7f87e6ae3ec0 -1 Falling back to public interface 2019-12-21T16:04:14.358 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:04:14.356+0000 7f87e6ae3ec0 -1 osd.1 51 log_to_monitors {default=true} 2019-12-21T16:04:14.558 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i++ )) 2019-12-21T16:04:14.559 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i < 300 )) 2019-12-21T16:04:14.559 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: echo 1 2019-12-21T16:04:14.559 INFO:tasks.workunit.client.0.smithi172.stderr:1 2019-12-21T16:04:14.559 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: ceph osd dump 2019-12-21T16:04:14.559 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: grep 'osd.1 up' 2019-12-21T16:04:14.791 INFO:tasks.workunit.client.0.smithi172.stderr:osd.1 up in weight 1 up_from 55 up_thru 55 down_at 52 last_clean_interval [41,51) [v2:127.0.0.1:6810/152467,v1:127.0.0.1:6811/152467] [v2:127.0.0.1:6812/152467,v1:127.0.0.1:6813/152467] exists,up 26b5b539-7134-484d-9f48-beaa93e824d7 2019-12-21T16:04:14.791 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=0 2019-12-21T16:04:14.791 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: break 2019-12-21T16:04:14.791 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: return 0 2019-12-21T16:04:14.792 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1239: objectstore_tool: wait_for_clean 2019-12-21T16:04:14.792 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1548: wait_for_clean: local cmd= 2019-12-21T16:04:14.792 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1549: wait_for_clean: local num_active_clean=-1 2019-12-21T16:04:14.792 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1550: wait_for_clean: local cur_active_clean 2019-12-21T16:04:14.792 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1551: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2019-12-21T16:04:14.792 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1551: wait_for_clean: get_timeout_delays 90 .1 2019-12-21T16:04:14.792 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1494: get_timeout_delays: shopt -q -o xtrace 2019-12-21T16:04:14.793 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1494: get_timeout_delays: echo true 2019-12-21T16:04:14.793 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1494: get_timeout_delays: local trace=true 2019-12-21T16:04:14.793 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1495: get_timeout_delays: true 2019-12-21T16:04:14.793 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1495: get_timeout_delays: shopt -u -o xtrace 2019-12-21T16:04:14.883 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1551: wait_for_clean: local -a delays 2019-12-21T16:04:14.883 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1552: wait_for_clean: local -i loop=0 2019-12-21T16:04:14.883 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1554: wait_for_clean: flush_pg_stats 2019-12-21T16:04:14.883 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1977: flush_pg_stats: local timeout=300 2019-12-21T16:04:14.884 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1979: flush_pg_stats: ceph osd ls 2019-12-21T16:04:15.135 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1979: flush_pg_stats: ids='0 2019-12-21T16:04:15.135 INFO:tasks.workunit.client.0.smithi172.stderr:1 2019-12-21T16:04:15.135 INFO:tasks.workunit.client.0.smithi172.stderr:2' 2019-12-21T16:04:15.135 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1980: flush_pg_stats: seqs= 2019-12-21T16:04:15.136 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: flush_pg_stats: for osd in $ids 2019-12-21T16:04:15.136 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2019-12-21T16:04:15.243 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: seq=21474836495 2019-12-21T16:04:15.243 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: flush_pg_stats: seqs=' 0-21474836495' 2019-12-21T16:04:15.243 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: flush_pg_stats: for osd in $ids 2019-12-21T16:04:15.244 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2019-12-21T16:04:15.354 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: seq=236223201282 2019-12-21T16:04:15.354 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: flush_pg_stats: seqs=' 0-21474836495 1-236223201282' 2019-12-21T16:04:15.354 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: flush_pg_stats: for osd in $ids 2019-12-21T16:04:15.355 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2019-12-21T16:04:15.464 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: seq=73014444044 2019-12-21T16:04:15.464 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: flush_pg_stats: seqs=' 0-21474836495 1-236223201282 2-73014444044' 2019-12-21T16:04:15.464 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: flush_pg_stats: for s in $seqs 2019-12-21T16:04:15.465 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: echo 0-21474836495 2019-12-21T16:04:15.465 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: cut -d - -f 1 2019-12-21T16:04:15.465 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: osd=0 2019-12-21T16:04:15.466 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: echo 0-21474836495 2019-12-21T16:04:15.466 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: cut -d - -f 2 2019-12-21T16:04:15.466 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: seq=21474836495 2019-12-21T16:04:15.466 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1989: flush_pg_stats: echo 'waiting osd.0 seq 21474836495' 2019-12-21T16:04:15.467 INFO:tasks.workunit.client.0.smithi172.stderr:waiting osd.0 seq 21474836495 2019-12-21T16:04:15.467 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 0 2019-12-21T16:04:15.694 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 21474836494 -lt 21474836495 2019-12-21T16:04:15.694 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1991: flush_pg_stats: sleep 1 2019-12-21T16:04:16.695 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1992: flush_pg_stats: '[' 300 -eq 0 ']' 2019-12-21T16:04:16.696 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 0 2019-12-21T16:04:16.931 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 21474836495 -lt 21474836495 2019-12-21T16:04:16.931 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: flush_pg_stats: for s in $seqs 2019-12-21T16:04:16.931 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: echo 1-236223201282 2019-12-21T16:04:16.932 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: cut -d - -f 1 2019-12-21T16:04:16.932 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: osd=1 2019-12-21T16:04:16.933 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: echo 1-236223201282 2019-12-21T16:04:16.933 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: cut -d - -f 2 2019-12-21T16:04:16.934 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: seq=236223201282 2019-12-21T16:04:16.934 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1989: flush_pg_stats: echo 'waiting osd.1 seq 236223201282' 2019-12-21T16:04:16.934 INFO:tasks.workunit.client.0.smithi172.stderr:waiting osd.1 seq 236223201282 2019-12-21T16:04:16.934 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 1 2019-12-21T16:04:17.160 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 236223201282 -lt 236223201282 2019-12-21T16:04:17.160 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: flush_pg_stats: for s in $seqs 2019-12-21T16:04:17.161 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: echo 2-73014444044 2019-12-21T16:04:17.161 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: cut -d - -f 1 2019-12-21T16:04:17.162 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: osd=2 2019-12-21T16:04:17.162 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: echo 2-73014444044 2019-12-21T16:04:17.162 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: cut -d - -f 2 2019-12-21T16:04:17.163 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: seq=73014444044 2019-12-21T16:04:17.163 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1989: flush_pg_stats: echo 'waiting osd.2 seq 73014444044' 2019-12-21T16:04:17.163 INFO:tasks.workunit.client.0.smithi172.stderr:waiting osd.2 seq 73014444044 2019-12-21T16:04:17.163 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 2 2019-12-21T16:04:17.391 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 73014444044 -lt 73014444044 2019-12-21T16:04:17.391 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1555: wait_for_clean: get_num_pgs 2019-12-21T16:04:17.392 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: ceph --format json status 2019-12-21T16:04:17.392 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: jq .pgmap.num_pgs 2019-12-21T16:04:17.659 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1555: wait_for_clean: test 5 == 0 2019-12-21T16:04:17.659 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1559: wait_for_clean: true 2019-12-21T16:04:17.659 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1563: wait_for_clean: get_num_active_clean 2019-12-21T16:04:17.660 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: get_num_active_clean: local expression 2019-12-21T16:04:17.660 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2019-12-21T16:04:17.660 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("stale") | not)' 2019-12-21T16:04:17.660 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: ceph --format json pg dump pgs 2019-12-21T16:04:17.660 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2019-12-21T16:04:17.881 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1563: wait_for_clean: cur_active_clean=5 2019-12-21T16:04:17.882 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1564: wait_for_clean: get_num_pgs 2019-12-21T16:04:17.882 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: ceph --format json status 2019-12-21T16:04:17.882 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: jq .pgmap.num_pgs 2019-12-21T16:04:18.167 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1564: wait_for_clean: test 5 = 5 2019-12-21T16:04:18.167 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1564: wait_for_clean: break 2019-12-21T16:04:18.167 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: return 0 2019-12-21T16:04:18.168 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:315: auto_repair_erasure_coded: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2019-12-21T16:04:18.198 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:316: auto_repair_erasure_coded: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2019-12-21T16:04:18.199 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:319: auto_repair_erasure_coded: teardown td/osd-scrub-repair 2019-12-21T16:04:18.200 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2019-12-21T16:04:18.200 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2019-12-21T16:04:18.200 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2019-12-21T16:04:18.200 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2019-12-21T16:04:18.200 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2019-12-21T16:04:18.200 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2019-12-21T16:04:18.201 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2019-12-21T16:04:18.201 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2019-12-21T16:04:18.308 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2019-12-21T16:04:18.308 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2019-12-21T16:04:18.309 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2019-12-21T16:04:18.309 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2019-12-21T16:04:18.310 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2019-12-21T16:04:18.310 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2019-12-21T16:04:18.310 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2019-12-21T16:04:18.312 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local 'pattern=|/usr/lib/systemd/systemd-coredump %P %u %g %s %t %c %h %e' 2019-12-21T16:04:18.312 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' '|' = '|' ']' 2019-12-21T16:04:18.312 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:177: teardown: pattern= 2019-12-21T16:04:18.312 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2019-12-21T16:04:18.313 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname '' 2019-12-21T16:04:18.313 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls . 2019-12-21T16:04:18.314 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2019-12-21T16:04:18.314 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2019-12-21T16:04:18.321 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2019-12-21T16:04:18.321 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T16:04:18.321 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.119575 2019-12-21T16:04:18.321 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.119575 2019-12-21T16:04:18.323 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2019-12-21T16:04:18.324 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2019-12-21T16:04:18.324 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:62: run: for func in $funcs 2019-12-21T16:04:18.324 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: TEST_corrupt_and_repair_jerasure_appends td/osd-scrub-repair 2019-12-21T16:04:18.324 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:734: TEST_corrupt_and_repair_jerasure_appends: corrupt_and_repair_jerasure td/osd-scrub-repair false 2019-12-21T16:04:18.325 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:710: corrupt_and_repair_jerasure: local dir=td/osd-scrub-repair 2019-12-21T16:04:18.325 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:711: corrupt_and_repair_jerasure: local allow_overwrites=false 2019-12-21T16:04:18.326 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:712: corrupt_and_repair_jerasure: local poolname=ecpool 2019-12-21T16:04:18.326 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:714: corrupt_and_repair_jerasure: setup td/osd-scrub-repair 2019-12-21T16:04:18.326 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2019-12-21T16:04:18.326 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2019-12-21T16:04:18.326 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2019-12-21T16:04:18.326 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2019-12-21T16:04:18.326 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2019-12-21T16:04:18.327 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2019-12-21T16:04:18.327 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2019-12-21T16:04:18.327 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2019-12-21T16:04:18.327 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2019-12-21T16:04:18.327 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2019-12-21T16:04:18.329 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2019-12-21T16:04:18.329 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2019-12-21T16:04:18.330 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2019-12-21T16:04:18.330 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2019-12-21T16:04:18.330 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2019-12-21T16:04:18.330 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2019-12-21T16:04:18.330 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2019-12-21T16:04:18.330 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local 'pattern=|/usr/lib/systemd/systemd-coredump %P %u %g %s %t %c %h %e' 2019-12-21T16:04:18.330 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' '|' = '|' ']' 2019-12-21T16:04:18.331 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:177: teardown: pattern= 2019-12-21T16:04:18.331 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2019-12-21T16:04:18.331 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname '' 2019-12-21T16:04:18.331 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls . 2019-12-21T16:04:18.332 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2019-12-21T16:04:18.332 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2019-12-21T16:04:18.332 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2019-12-21T16:04:18.332 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T16:04:18.333 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.119575 2019-12-21T16:04:18.333 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.119575 2019-12-21T16:04:18.333 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2019-12-21T16:04:18.333 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2019-12-21T16:04:18.333 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2019-12-21T16:04:18.334 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2019-12-21T16:04:18.334 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T16:04:18.334 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.119575 2019-12-21T16:04:18.334 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.119575 2019-12-21T16:04:18.336 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2019-12-21T16:04:18.336 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2019-12-21T16:04:18.337 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2019-12-21T16:04:18.337 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2019-12-21T16:04:18.337 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:715: corrupt_and_repair_jerasure: run_mon td/osd-scrub-repair a 2019-12-21T16:04:18.339 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2019-12-21T16:04:18.339 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2019-12-21T16:04:18.339 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2019-12-21T16:04:18.339 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2019-12-21T16:04:18.340 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2019-12-21T16:04:18.340 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2019-12-21T16:04:18.626 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2019-12-21T16:04:18.626 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2019-12-21T16:04:18.627 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2019-12-21T16:04:18.627 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2019-12-21T16:04:18.627 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T16:04:18.627 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.119575 2019-12-21T16:04:18.627 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.119575/$cluster-$name.asok' 2019-12-21T16:04:18.627 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.119575/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 2019-12-21T16:04:18.760 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: cat 2019-12-21T16:04:18.761 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: get_config mon a fsid 2019-12-21T16:04:18.761 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: get_config: local daemon=mon 2019-12-21T16:04:18.762 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: get_config: local id=a 2019-12-21T16:04:18.762 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local config=fsid 2019-12-21T16:04:18.762 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: jq -r .fsid 2019-12-21T16:04:18.762 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: get_asok_path mon.a 2019-12-21T16:04:18.762 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2019-12-21T16:04:18.763 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2019-12-21T16:04:18.763 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T16:04:18.764 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T16:04:18.764 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.119575 2019-12-21T16:04:18.764 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.119575/ceph-mon.a.asok 2019-12-21T16:04:18.764 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: CEPH_ARGS= 2019-12-21T16:04:18.764 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: ceph --format json daemon /tmp/ceph-asok.119575/ceph-mon.a.asok config get fsid 2019-12-21T16:04:18.850 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: get_config mon a mon_host 2019-12-21T16:04:18.850 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: get_config: local daemon=mon 2019-12-21T16:04:18.850 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: get_config: local id=a 2019-12-21T16:04:18.850 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local config=mon_host 2019-12-21T16:04:18.850 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: jq -r .mon_host 2019-12-21T16:04:18.851 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: get_asok_path mon.a 2019-12-21T16:04:18.851 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2019-12-21T16:04:18.851 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2019-12-21T16:04:18.851 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T16:04:18.851 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T16:04:18.851 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.119575 2019-12-21T16:04:18.851 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.119575/ceph-mon.a.asok 2019-12-21T16:04:18.852 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: CEPH_ARGS= 2019-12-21T16:04:18.852 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: ceph --format json daemon /tmp/ceph-asok.119575/ceph-mon.a.asok config get mon_host 2019-12-21T16:04:18.937 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:716: corrupt_and_repair_jerasure: run_mgr td/osd-scrub-repair x 2019-12-21T16:04:18.938 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:551: run_mgr: local dir=td/osd-scrub-repair 2019-12-21T16:04:18.938 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: shift 2019-12-21T16:04:18.938 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local id=x 2019-12-21T16:04:18.938 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2019-12-21T16:04:18.938 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local data=td/osd-scrub-repair/x 2019-12-21T16:04:18.938 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:558: run_mgr: get_asok_path 2019-12-21T16:04:18.939 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2019-12-21T16:04:18.939 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2019-12-21T16:04:18.939 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2019-12-21T16:04:18.939 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T16:04:18.939 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.119575 2019-12-21T16:04:18.939 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.119575/$cluster-$name.asok' 2019-12-21T16:04:18.940 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:558: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2019-12-21T16:04:18.940 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:558: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.119575/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2019-12-21T16:04:19.561 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:717: corrupt_and_repair_jerasure: seq 0 3 2019-12-21T16:04:19.565 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:717: corrupt_and_repair_jerasure: for id in $(seq 0 3) 2019-12-21T16:04:19.565 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:718: corrupt_and_repair_jerasure: '[' false = true ']' 2019-12-21T16:04:19.565 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:721: corrupt_and_repair_jerasure: run_osd_filestore td/osd-scrub-repair 0 2019-12-21T16:04:19.565 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:685: run_osd_filestore: local dir=td/osd-scrub-repair 2019-12-21T16:04:19.565 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:686: run_osd_filestore: shift 2019-12-21T16:04:19.566 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:687: run_osd_filestore: local id=0 2019-12-21T16:04:19.566 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:688: run_osd_filestore: shift 2019-12-21T16:04:19.566 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:689: run_osd_filestore: local osd_data=td/osd-scrub-repair/0 2019-12-21T16:04:19.566 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:691: run_osd_filestore: local 'ceph_args=--fsid=973e3fc1-e63e-4bde-9db6-84c68fd5e93e --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2019-12-21T16:04:19.566 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:692: run_osd_filestore: ceph_args+=' --osd-failsafe-full-ratio=.99' 2019-12-21T16:04:19.566 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:693: run_osd_filestore: ceph_args+=' --osd-journal-size=100' 2019-12-21T16:04:19.566 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:694: run_osd_filestore: ceph_args+=' --osd-scrub-load-threshold=2000' 2019-12-21T16:04:19.567 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:695: run_osd_filestore: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2019-12-21T16:04:19.567 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:696: run_osd_filestore: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2019-12-21T16:04:19.567 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:697: run_osd_filestore: ceph_args+=' --chdir=' 2019-12-21T16:04:19.567 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:698: run_osd_filestore: ceph_args+= 2019-12-21T16:04:19.567 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:699: run_osd_filestore: ceph_args+=' --run-dir=td/osd-scrub-repair' 2019-12-21T16:04:19.568 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:700: run_osd_filestore: get_asok_path 2019-12-21T16:04:19.568 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2019-12-21T16:04:19.568 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2019-12-21T16:04:19.568 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2019-12-21T16:04:19.568 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T16:04:19.568 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.119575 2019-12-21T16:04:19.568 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.119575/$cluster-$name.asok' 2019-12-21T16:04:19.569 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:700: run_osd_filestore: ceph_args+=' --admin-socket=/tmp/ceph-asok.119575/$cluster-$name.asok' 2019-12-21T16:04:19.569 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:701: run_osd_filestore: ceph_args+=' --debug-osd=20' 2019-12-21T16:04:19.569 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:702: run_osd_filestore: ceph_args+=' --debug-ms=1' 2019-12-21T16:04:19.569 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:703: run_osd_filestore: ceph_args+=' --debug-monc=20' 2019-12-21T16:04:19.569 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:704: run_osd_filestore: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2019-12-21T16:04:19.569 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:705: run_osd_filestore: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2019-12-21T16:04:19.569 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:706: run_osd_filestore: ceph_args+=' --osd-max-object-name-len=460' 2019-12-21T16:04:19.570 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:707: run_osd_filestore: ceph_args+=' --osd-max-object-namespace-len=64' 2019-12-21T16:04:19.570 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:708: run_osd_filestore: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2019-12-21T16:04:19.570 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:709: run_osd_filestore: ceph_args+=' ' 2019-12-21T16:04:19.570 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:710: run_osd_filestore: ceph_args+= 2019-12-21T16:04:19.570 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:711: run_osd_filestore: mkdir -p td/osd-scrub-repair/0 2019-12-21T16:04:19.570 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:713: run_osd_filestore: uuidgen 2019-12-21T16:04:19.571 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:713: run_osd_filestore: local uuid=30ee7539-61e8-47d6-98f1-2df51b8e8440 2019-12-21T16:04:19.572 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:714: run_osd_filestore: echo 'add osd2 30ee7539-61e8-47d6-98f1-2df51b8e8440' 2019-12-21T16:04:19.572 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:715: run_osd_filestore: ceph-authtool --gen-print-key 2019-12-21T16:04:19.572 INFO:tasks.workunit.client.0.smithi172.stdout:add osd2 30ee7539-61e8-47d6-98f1-2df51b8e8440 2019-12-21T16:04:19.579 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:715: run_osd_filestore: OSD_SECRET=AQCDQv5dypNsIhAA2tti8W1f6PPgotImu6XVeA== 2019-12-21T16:04:19.579 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:716: run_osd_filestore: echo '{"cephx_secret": "AQCDQv5dypNsIhAA2tti8W1f6PPgotImu6XVeA=="}' 2019-12-21T16:04:19.579 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:717: run_osd_filestore: ceph osd new 30ee7539-61e8-47d6-98f1-2df51b8e8440 -i td/osd-scrub-repair/0/new.json 2019-12-21T16:04:19.852 INFO:tasks.workunit.client.0.smithi172.stdout:0 2019-12-21T16:04:19.861 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:718: run_osd_filestore: rm td/osd-scrub-repair/0/new.json 2019-12-21T16:04:19.862 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:719: run_osd_filestore: ceph-osd -i 0 --fsid=973e3fc1-e63e-4bde-9db6-84c68fd5e93e --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.119575/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCDQv5dypNsIhAA2tti8W1f6PPgotImu6XVeA== --osd-uuid 30ee7539-61e8-47d6-98f1-2df51b8e8440 --osd-objectstore=filestore 2019-12-21T16:04:19.883 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:04:19.882+0000 7fc0e3540ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T16:04:19.885 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:04:19.884+0000 7fc0e3540ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T16:04:19.888 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:04:19.886+0000 7fc0e3540ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T16:04:20.228 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:04:20.227+0000 7fc0e3540ec0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2019-12-21T16:04:20.556 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:04:20.554+0000 7fc0e3540ec0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2019-12-21T16:04:20.556 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:04:20.555+0000 7fc0e3540ec0 -1 journal do_read_entry(4096): bad header magic 2019-12-21T16:04:20.557 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:04:20.555+0000 7fc0e3540ec0 -1 journal do_read_entry(4096): bad header magic 2019-12-21T16:04:20.982 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:721: run_osd_filestore: local key_fn=td/osd-scrub-repair/0/keyring 2019-12-21T16:04:20.983 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:722: run_osd_filestore: cat 2019-12-21T16:04:20.983 INFO:tasks.workunit.client.0.smithi172.stdout:adding osd0 key to auth repository 2019-12-21T16:04:20.984 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:726: run_osd_filestore: echo adding osd0 key to auth repository 2019-12-21T16:04:20.984 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:727: run_osd_filestore: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2019-12-21T16:04:21.156 INFO:tasks.workunit.client.0.smithi172.stderr:Error EINVAL: key for osd.0 not found in provided keyring 2019-12-21T16:04:21.164 INFO:tasks.workunit.client.0.smithi172.stdout:start osd.0 2019-12-21T16:04:21.164 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:728: run_osd_filestore: echo start osd.0 2019-12-21T16:04:21.165 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:729: run_osd_filestore: ceph-osd -i 0 --fsid=973e3fc1-e63e-4bde-9db6-84c68fd5e93e --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.119575/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2019-12-21T16:04:21.165 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:732: run_osd_filestore: ceph osd dump --format=json 2019-12-21T16:04:21.165 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:732: run_osd_filestore: jq '.flags_set[]' 2019-12-21T16:04:21.165 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:732: run_osd_filestore: grep -q '"noup"' 2019-12-21T16:04:21.184 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:04:21.182+0000 7fc246650ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T16:04:21.186 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:04:21.184+0000 7fc246650ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T16:04:21.187 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:04:21.185+0000 7fc246650ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T16:04:21.189 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:04:21.187+0000 7fc246650ec0 -1 Falling back to public interface 2019-12-21T16:04:21.430 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:735: run_osd_filestore: wait_for_osd up 0 2019-12-21T16:04:21.431 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:914: wait_for_osd: local state=up 2019-12-21T16:04:21.431 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local id=0 2019-12-21T16:04:21.431 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: status=1 2019-12-21T16:04:21.431 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i=0 )) 2019-12-21T16:04:21.431 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i < 300 )) 2019-12-21T16:04:21.431 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: echo 0 2019-12-21T16:04:21.432 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: ceph osd dump 2019-12-21T16:04:21.432 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: grep 'osd.0 up' 2019-12-21T16:04:21.432 INFO:tasks.workunit.client.0.smithi172.stdout:0 2019-12-21T16:04:21.552 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:04:21.551+0000 7fc246650ec0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2019-12-21T16:04:21.560 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:04:21.559+0000 7fc246650ec0 -1 journal do_read_entry(8192): bad header magic 2019-12-21T16:04:21.561 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:04:21.559+0000 7fc246650ec0 -1 journal do_read_entry(8192): bad header magic 2019-12-21T16:04:21.567 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:04:21.565+0000 7fc246650ec0 -1 osd.0 0 log_to_monitors {default=true} 2019-12-21T16:04:21.613 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: sleep 1 2019-12-21T16:04:22.614 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i++ )) 2019-12-21T16:04:22.615 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i < 300 )) 2019-12-21T16:04:22.615 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: echo 1 2019-12-21T16:04:22.615 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: ceph osd dump 2019-12-21T16:04:22.615 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: grep 'osd.0 up' 2019-12-21T16:04:22.615 INFO:tasks.workunit.client.0.smithi172.stdout:1 2019-12-21T16:04:22.789 INFO:tasks.workunit.client.0.smithi172.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/153395,v1:127.0.0.1:6801/153395] [v2:127.0.0.1:6802/153395,v1:127.0.0.1:6803/153395] exists,up 30ee7539-61e8-47d6-98f1-2df51b8e8440 2019-12-21T16:04:22.789 INFO:tasks.workunit.client.0.smithi172.stdout:add osd2 627d9f35-2c70-4ea0-8821-1ca4dc71496b 2019-12-21T16:04:22.790 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=0 2019-12-21T16:04:22.790 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: break 2019-12-21T16:04:22.790 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: return 0 2019-12-21T16:04:22.790 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:717: corrupt_and_repair_jerasure: for id in $(seq 0 3) 2019-12-21T16:04:22.790 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:718: corrupt_and_repair_jerasure: '[' false = true ']' 2019-12-21T16:04:22.790 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:721: corrupt_and_repair_jerasure: run_osd_filestore td/osd-scrub-repair 1 2019-12-21T16:04:22.791 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:685: run_osd_filestore: local dir=td/osd-scrub-repair 2019-12-21T16:04:22.791 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:686: run_osd_filestore: shift 2019-12-21T16:04:22.791 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:687: run_osd_filestore: local id=1 2019-12-21T16:04:22.791 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:688: run_osd_filestore: shift 2019-12-21T16:04:22.791 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:689: run_osd_filestore: local osd_data=td/osd-scrub-repair/1 2019-12-21T16:04:22.791 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:691: run_osd_filestore: local 'ceph_args=--fsid=973e3fc1-e63e-4bde-9db6-84c68fd5e93e --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2019-12-21T16:04:22.792 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:692: run_osd_filestore: ceph_args+=' --osd-failsafe-full-ratio=.99' 2019-12-21T16:04:22.792 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:693: run_osd_filestore: ceph_args+=' --osd-journal-size=100' 2019-12-21T16:04:22.792 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:694: run_osd_filestore: ceph_args+=' --osd-scrub-load-threshold=2000' 2019-12-21T16:04:22.792 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:695: run_osd_filestore: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2019-12-21T16:04:22.792 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:696: run_osd_filestore: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2019-12-21T16:04:22.792 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:697: run_osd_filestore: ceph_args+=' --chdir=' 2019-12-21T16:04:22.792 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:698: run_osd_filestore: ceph_args+= 2019-12-21T16:04:22.793 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:699: run_osd_filestore: ceph_args+=' --run-dir=td/osd-scrub-repair' 2019-12-21T16:04:22.793 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:700: run_osd_filestore: get_asok_path 2019-12-21T16:04:22.793 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2019-12-21T16:04:22.793 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2019-12-21T16:04:22.793 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2019-12-21T16:04:22.793 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T16:04:22.793 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.119575 2019-12-21T16:04:22.794 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.119575/$cluster-$name.asok' 2019-12-21T16:04:22.794 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:700: run_osd_filestore: ceph_args+=' --admin-socket=/tmp/ceph-asok.119575/$cluster-$name.asok' 2019-12-21T16:04:22.794 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:701: run_osd_filestore: ceph_args+=' --debug-osd=20' 2019-12-21T16:04:22.794 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:702: run_osd_filestore: ceph_args+=' --debug-ms=1' 2019-12-21T16:04:22.794 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:703: run_osd_filestore: ceph_args+=' --debug-monc=20' 2019-12-21T16:04:22.794 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:704: run_osd_filestore: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2019-12-21T16:04:22.794 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:705: run_osd_filestore: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2019-12-21T16:04:22.795 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:706: run_osd_filestore: ceph_args+=' --osd-max-object-name-len=460' 2019-12-21T16:04:22.795 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:707: run_osd_filestore: ceph_args+=' --osd-max-object-namespace-len=64' 2019-12-21T16:04:22.795 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:708: run_osd_filestore: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2019-12-21T16:04:22.795 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:709: run_osd_filestore: ceph_args+=' ' 2019-12-21T16:04:22.795 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:710: run_osd_filestore: ceph_args+= 2019-12-21T16:04:22.795 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:711: run_osd_filestore: mkdir -p td/osd-scrub-repair/1 2019-12-21T16:04:22.795 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:713: run_osd_filestore: uuidgen 2019-12-21T16:04:22.796 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:713: run_osd_filestore: local uuid=627d9f35-2c70-4ea0-8821-1ca4dc71496b 2019-12-21T16:04:22.796 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:714: run_osd_filestore: echo 'add osd2 627d9f35-2c70-4ea0-8821-1ca4dc71496b' 2019-12-21T16:04:22.796 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:715: run_osd_filestore: ceph-authtool --gen-print-key 2019-12-21T16:04:22.802 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:715: run_osd_filestore: OSD_SECRET=AQCGQv5dM6a2LxAA80HrWiFbf9AgbcruQFnW1w== 2019-12-21T16:04:22.802 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:716: run_osd_filestore: echo '{"cephx_secret": "AQCGQv5dM6a2LxAA80HrWiFbf9AgbcruQFnW1w=="}' 2019-12-21T16:04:22.802 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:717: run_osd_filestore: ceph osd new 627d9f35-2c70-4ea0-8821-1ca4dc71496b -i td/osd-scrub-repair/1/new.json 2019-12-21T16:04:23.133 INFO:tasks.workunit.client.0.smithi172.stdout:1 2019-12-21T16:04:23.142 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:718: run_osd_filestore: rm td/osd-scrub-repair/1/new.json 2019-12-21T16:04:23.143 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:719: run_osd_filestore: ceph-osd -i 1 --fsid=973e3fc1-e63e-4bde-9db6-84c68fd5e93e --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.119575/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCGQv5dM6a2LxAA80HrWiFbf9AgbcruQFnW1w== --osd-uuid 627d9f35-2c70-4ea0-8821-1ca4dc71496b --osd-objectstore=filestore 2019-12-21T16:04:23.165 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:04:23.163+0000 7f17f7ba1ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T16:04:23.183 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:04:23.181+0000 7f17f7ba1ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T16:04:23.185 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:04:23.183+0000 7f17f7ba1ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T16:04:23.651 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:04:23.650+0000 7f17f7ba1ec0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2019-12-21T16:04:24.020 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:04:24.019+0000 7f17f7ba1ec0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2019-12-21T16:04:24.021 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:04:24.019+0000 7f17f7ba1ec0 -1 journal do_read_entry(4096): bad header magic 2019-12-21T16:04:24.021 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:04:24.019+0000 7f17f7ba1ec0 -1 journal do_read_entry(4096): bad header magic 2019-12-21T16:04:24.437 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:721: run_osd_filestore: local key_fn=td/osd-scrub-repair/1/keyring 2019-12-21T16:04:24.437 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:722: run_osd_filestore: cat 2019-12-21T16:04:24.438 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:726: run_osd_filestore: echo adding osd1 key to auth repository 2019-12-21T16:04:24.438 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:727: run_osd_filestore: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2019-12-21T16:04:24.439 INFO:tasks.workunit.client.0.smithi172.stdout:adding osd1 key to auth repository 2019-12-21T16:04:24.704 INFO:tasks.workunit.client.0.smithi172.stderr:Error EINVAL: key for osd.1 not found in provided keyring 2019-12-21T16:04:24.711 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:728: run_osd_filestore: echo start osd.1 2019-12-21T16:04:24.712 INFO:tasks.workunit.client.0.smithi172.stdout:start osd.1 2019-12-21T16:04:24.712 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:729: run_osd_filestore: ceph-osd -i 1 --fsid=973e3fc1-e63e-4bde-9db6-84c68fd5e93e --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.119575/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2019-12-21T16:04:24.712 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:732: run_osd_filestore: ceph osd dump --format=json 2019-12-21T16:04:24.713 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:732: run_osd_filestore: jq '.flags_set[]' 2019-12-21T16:04:24.713 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:732: run_osd_filestore: grep -q '"noup"' 2019-12-21T16:04:24.730 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:04:24.729+0000 7f7f2431fec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T16:04:24.732 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:04:24.731+0000 7f7f2431fec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T16:04:24.734 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:04:24.733+0000 7f7f2431fec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T16:04:24.735 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:04:24.734+0000 7f7f2431fec0 -1 Falling back to public interface 2019-12-21T16:04:25.023 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:735: run_osd_filestore: wait_for_osd up 1 2019-12-21T16:04:25.023 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:914: wait_for_osd: local state=up 2019-12-21T16:04:25.023 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local id=1 2019-12-21T16:04:25.024 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: status=1 2019-12-21T16:04:25.024 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i=0 )) 2019-12-21T16:04:25.024 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i < 300 )) 2019-12-21T16:04:25.024 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: echo 0 2019-12-21T16:04:25.024 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: ceph osd dump 2019-12-21T16:04:25.024 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: grep 'osd.1 up' 2019-12-21T16:04:25.025 INFO:tasks.workunit.client.0.smithi172.stdout:0 2019-12-21T16:04:25.025 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:04:25.023+0000 7f7f2431fec0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2019-12-21T16:04:25.041 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:04:25.040+0000 7f7f2431fec0 -1 journal do_read_entry(8192): bad header magic 2019-12-21T16:04:25.042 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:04:25.040+0000 7f7f2431fec0 -1 journal do_read_entry(8192): bad header magic 2019-12-21T16:04:25.048 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:04:25.046+0000 7f7f2431fec0 -1 osd.1 0 log_to_monitors {default=true} 2019-12-21T16:04:25.294 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: sleep 1 2019-12-21T16:04:26.295 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i++ )) 2019-12-21T16:04:26.296 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i < 300 )) 2019-12-21T16:04:26.296 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: echo 1 2019-12-21T16:04:26.296 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: ceph osd dump 2019-12-21T16:04:26.296 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: grep 'osd.1 up' 2019-12-21T16:04:26.296 INFO:tasks.workunit.client.0.smithi172.stdout:1 2019-12-21T16:04:26.531 INFO:tasks.workunit.client.0.smithi172.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/153733,v1:127.0.0.1:6811/153733] [v2:127.0.0.1:6812/153733,v1:127.0.0.1:6813/153733] exists,up 627d9f35-2c70-4ea0-8821-1ca4dc71496b 2019-12-21T16:04:26.531 INFO:tasks.workunit.client.0.smithi172.stdout:add osd2 f6c07742-f5aa-4b9f-b1b9-38cac5d974cc 2019-12-21T16:04:26.531 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=0 2019-12-21T16:04:26.532 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: break 2019-12-21T16:04:26.532 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: return 0 2019-12-21T16:04:26.532 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:717: corrupt_and_repair_jerasure: for id in $(seq 0 3) 2019-12-21T16:04:26.532 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:718: corrupt_and_repair_jerasure: '[' false = true ']' 2019-12-21T16:04:26.532 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:721: corrupt_and_repair_jerasure: run_osd_filestore td/osd-scrub-repair 2 2019-12-21T16:04:26.532 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:685: run_osd_filestore: local dir=td/osd-scrub-repair 2019-12-21T16:04:26.532 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:686: run_osd_filestore: shift 2019-12-21T16:04:26.533 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:687: run_osd_filestore: local id=2 2019-12-21T16:04:26.533 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:688: run_osd_filestore: shift 2019-12-21T16:04:26.533 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:689: run_osd_filestore: local osd_data=td/osd-scrub-repair/2 2019-12-21T16:04:26.533 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:691: run_osd_filestore: local 'ceph_args=--fsid=973e3fc1-e63e-4bde-9db6-84c68fd5e93e --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2019-12-21T16:04:26.533 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:692: run_osd_filestore: ceph_args+=' --osd-failsafe-full-ratio=.99' 2019-12-21T16:04:26.533 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:693: run_osd_filestore: ceph_args+=' --osd-journal-size=100' 2019-12-21T16:04:26.533 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:694: run_osd_filestore: ceph_args+=' --osd-scrub-load-threshold=2000' 2019-12-21T16:04:26.534 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:695: run_osd_filestore: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2019-12-21T16:04:26.534 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:696: run_osd_filestore: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2019-12-21T16:04:26.534 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:697: run_osd_filestore: ceph_args+=' --chdir=' 2019-12-21T16:04:26.534 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:698: run_osd_filestore: ceph_args+= 2019-12-21T16:04:26.534 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:699: run_osd_filestore: ceph_args+=' --run-dir=td/osd-scrub-repair' 2019-12-21T16:04:26.534 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:700: run_osd_filestore: get_asok_path 2019-12-21T16:04:26.534 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2019-12-21T16:04:26.535 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2019-12-21T16:04:26.535 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2019-12-21T16:04:26.535 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T16:04:26.535 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.119575 2019-12-21T16:04:26.535 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.119575/$cluster-$name.asok' 2019-12-21T16:04:26.535 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:700: run_osd_filestore: ceph_args+=' --admin-socket=/tmp/ceph-asok.119575/$cluster-$name.asok' 2019-12-21T16:04:26.535 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:701: run_osd_filestore: ceph_args+=' --debug-osd=20' 2019-12-21T16:04:26.536 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:702: run_osd_filestore: ceph_args+=' --debug-ms=1' 2019-12-21T16:04:26.536 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:703: run_osd_filestore: ceph_args+=' --debug-monc=20' 2019-12-21T16:04:26.536 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:704: run_osd_filestore: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2019-12-21T16:04:26.536 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:705: run_osd_filestore: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2019-12-21T16:04:26.536 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:706: run_osd_filestore: ceph_args+=' --osd-max-object-name-len=460' 2019-12-21T16:04:26.536 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:707: run_osd_filestore: ceph_args+=' --osd-max-object-namespace-len=64' 2019-12-21T16:04:26.536 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:708: run_osd_filestore: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2019-12-21T16:04:26.537 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:709: run_osd_filestore: ceph_args+=' ' 2019-12-21T16:04:26.537 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:710: run_osd_filestore: ceph_args+= 2019-12-21T16:04:26.537 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:711: run_osd_filestore: mkdir -p td/osd-scrub-repair/2 2019-12-21T16:04:26.537 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:713: run_osd_filestore: uuidgen 2019-12-21T16:04:26.537 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:713: run_osd_filestore: local uuid=f6c07742-f5aa-4b9f-b1b9-38cac5d974cc 2019-12-21T16:04:26.537 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:714: run_osd_filestore: echo 'add osd2 f6c07742-f5aa-4b9f-b1b9-38cac5d974cc' 2019-12-21T16:04:26.537 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:715: run_osd_filestore: ceph-authtool --gen-print-key 2019-12-21T16:04:26.543 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:715: run_osd_filestore: OSD_SECRET=AQCKQv5d2bFMIBAAzgN3eaKi7tm5pzHMdQhCwA== 2019-12-21T16:04:26.544 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:716: run_osd_filestore: echo '{"cephx_secret": "AQCKQv5d2bFMIBAAzgN3eaKi7tm5pzHMdQhCwA=="}' 2019-12-21T16:04:26.544 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:717: run_osd_filestore: ceph osd new f6c07742-f5aa-4b9f-b1b9-38cac5d974cc -i td/osd-scrub-repair/2/new.json 2019-12-21T16:04:26.847 INFO:tasks.workunit.client.0.smithi172.stdout:2 2019-12-21T16:04:26.857 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:718: run_osd_filestore: rm td/osd-scrub-repair/2/new.json 2019-12-21T16:04:26.858 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:719: run_osd_filestore: ceph-osd -i 2 --fsid=973e3fc1-e63e-4bde-9db6-84c68fd5e93e --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.119575/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCKQv5d2bFMIBAAzgN3eaKi7tm5pzHMdQhCwA== --osd-uuid f6c07742-f5aa-4b9f-b1b9-38cac5d974cc --osd-objectstore=filestore 2019-12-21T16:04:26.879 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:04:26.878+0000 7f42f04abec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T16:04:26.881 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:04:26.879+0000 7f42f04abec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T16:04:26.883 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:04:26.881+0000 7f42f04abec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T16:04:27.241 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:04:27.239+0000 7f42f04abec0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2019-12-21T16:04:27.660 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:04:27.658+0000 7f42f04abec0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2019-12-21T16:04:27.660 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:04:27.659+0000 7f42f04abec0 -1 journal do_read_entry(4096): bad header magic 2019-12-21T16:04:27.661 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:04:27.659+0000 7f42f04abec0 -1 journal do_read_entry(4096): bad header magic 2019-12-21T16:04:28.093 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:721: run_osd_filestore: local key_fn=td/osd-scrub-repair/2/keyring 2019-12-21T16:04:28.094 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:722: run_osd_filestore: cat 2019-12-21T16:04:28.095 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:726: run_osd_filestore: echo adding osd2 key to auth repository 2019-12-21T16:04:28.095 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:727: run_osd_filestore: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2019-12-21T16:04:28.095 INFO:tasks.workunit.client.0.smithi172.stdout:adding osd2 key to auth repository 2019-12-21T16:04:28.378 INFO:tasks.workunit.client.0.smithi172.stdout:start osd.2 2019-12-21T16:04:28.379 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:728: run_osd_filestore: echo start osd.2 2019-12-21T16:04:28.379 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:729: run_osd_filestore: ceph-osd -i 2 --fsid=973e3fc1-e63e-4bde-9db6-84c68fd5e93e --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.119575/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2019-12-21T16:04:28.379 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:732: run_osd_filestore: jq '.flags_set[]' 2019-12-21T16:04:28.379 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:732: run_osd_filestore: grep -q '"noup"' 2019-12-21T16:04:28.379 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:732: run_osd_filestore: ceph osd dump --format=json 2019-12-21T16:04:28.397 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:04:28.395+0000 7f1fb4878ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T16:04:28.399 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:04:28.397+0000 7f1fb4878ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T16:04:28.402 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:04:28.399+0000 7f1fb4878ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T16:04:28.403 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:04:28.400+0000 7f1fb4878ec0 -1 Falling back to public interface 2019-12-21T16:04:28.706 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:04:28.703+0000 7f1fb4878ec0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2019-12-21T16:04:28.706 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:04:28.704+0000 7f1fb4878ec0 -1 journal do_read_entry(8192): bad header magic 2019-12-21T16:04:28.707 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:04:28.704+0000 7f1fb4878ec0 -1 journal do_read_entry(8192): bad header magic 2019-12-21T16:04:28.711 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:04:28.708+0000 7f1fb4878ec0 -1 osd.2 0 log_to_monitors {default=true} 2019-12-21T16:04:28.776 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:735: run_osd_filestore: wait_for_osd up 2 2019-12-21T16:04:28.776 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:914: wait_for_osd: local state=up 2019-12-21T16:04:28.776 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local id=2 2019-12-21T16:04:28.776 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: status=1 2019-12-21T16:04:28.776 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i=0 )) 2019-12-21T16:04:28.776 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i < 300 )) 2019-12-21T16:04:28.777 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: echo 0 2019-12-21T16:04:28.777 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: ceph osd dump 2019-12-21T16:04:28.777 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: grep 'osd.2 up' 2019-12-21T16:04:28.777 INFO:tasks.workunit.client.0.smithi172.stdout:0 2019-12-21T16:04:29.069 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: sleep 1 2019-12-21T16:04:30.071 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i++ )) 2019-12-21T16:04:30.071 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i < 300 )) 2019-12-21T16:04:30.071 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: echo 1 2019-12-21T16:04:30.071 INFO:tasks.workunit.client.0.smithi172.stdout:1 2019-12-21T16:04:30.072 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: ceph osd dump 2019-12-21T16:04:30.072 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: grep 'osd.2 up' 2019-12-21T16:04:30.305 INFO:tasks.workunit.client.0.smithi172.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/154032,v1:127.0.0.1:6819/154032] [v2:127.0.0.1:6820/154032,v1:127.0.0.1:6821/154032] exists,up f6c07742-f5aa-4b9f-b1b9-38cac5d974cc 2019-12-21T16:04:30.305 INFO:tasks.workunit.client.0.smithi172.stdout:add osd2 d2b8f48d-5699-449e-8fc7-707402a64a19 2019-12-21T16:04:30.305 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=0 2019-12-21T16:04:30.305 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: break 2019-12-21T16:04:30.305 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: return 0 2019-12-21T16:04:30.306 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:717: corrupt_and_repair_jerasure: for id in $(seq 0 3) 2019-12-21T16:04:30.306 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:718: corrupt_and_repair_jerasure: '[' false = true ']' 2019-12-21T16:04:30.306 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:721: corrupt_and_repair_jerasure: run_osd_filestore td/osd-scrub-repair 3 2019-12-21T16:04:30.306 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:685: run_osd_filestore: local dir=td/osd-scrub-repair 2019-12-21T16:04:30.306 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:686: run_osd_filestore: shift 2019-12-21T16:04:30.306 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:687: run_osd_filestore: local id=3 2019-12-21T16:04:30.306 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:688: run_osd_filestore: shift 2019-12-21T16:04:30.307 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:689: run_osd_filestore: local osd_data=td/osd-scrub-repair/3 2019-12-21T16:04:30.307 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:691: run_osd_filestore: local 'ceph_args=--fsid=973e3fc1-e63e-4bde-9db6-84c68fd5e93e --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2019-12-21T16:04:30.307 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:692: run_osd_filestore: ceph_args+=' --osd-failsafe-full-ratio=.99' 2019-12-21T16:04:30.307 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:693: run_osd_filestore: ceph_args+=' --osd-journal-size=100' 2019-12-21T16:04:30.307 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:694: run_osd_filestore: ceph_args+=' --osd-scrub-load-threshold=2000' 2019-12-21T16:04:30.307 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:695: run_osd_filestore: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2019-12-21T16:04:30.307 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:696: run_osd_filestore: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2019-12-21T16:04:30.308 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:697: run_osd_filestore: ceph_args+=' --chdir=' 2019-12-21T16:04:30.308 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:698: run_osd_filestore: ceph_args+= 2019-12-21T16:04:30.308 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:699: run_osd_filestore: ceph_args+=' --run-dir=td/osd-scrub-repair' 2019-12-21T16:04:30.308 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:700: run_osd_filestore: get_asok_path 2019-12-21T16:04:30.309 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2019-12-21T16:04:30.309 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2019-12-21T16:04:30.309 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2019-12-21T16:04:30.309 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T16:04:30.309 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.119575 2019-12-21T16:04:30.309 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.119575/$cluster-$name.asok' 2019-12-21T16:04:30.309 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:700: run_osd_filestore: ceph_args+=' --admin-socket=/tmp/ceph-asok.119575/$cluster-$name.asok' 2019-12-21T16:04:30.310 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:701: run_osd_filestore: ceph_args+=' --debug-osd=20' 2019-12-21T16:04:30.310 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:702: run_osd_filestore: ceph_args+=' --debug-ms=1' 2019-12-21T16:04:30.310 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:703: run_osd_filestore: ceph_args+=' --debug-monc=20' 2019-12-21T16:04:30.310 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:704: run_osd_filestore: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2019-12-21T16:04:30.310 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:705: run_osd_filestore: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2019-12-21T16:04:30.310 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:706: run_osd_filestore: ceph_args+=' --osd-max-object-name-len=460' 2019-12-21T16:04:30.310 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:707: run_osd_filestore: ceph_args+=' --osd-max-object-namespace-len=64' 2019-12-21T16:04:30.310 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:708: run_osd_filestore: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2019-12-21T16:04:30.311 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:709: run_osd_filestore: ceph_args+=' ' 2019-12-21T16:04:30.311 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:710: run_osd_filestore: ceph_args+= 2019-12-21T16:04:30.311 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:711: run_osd_filestore: mkdir -p td/osd-scrub-repair/3 2019-12-21T16:04:30.311 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:713: run_osd_filestore: uuidgen 2019-12-21T16:04:30.311 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:713: run_osd_filestore: local uuid=d2b8f48d-5699-449e-8fc7-707402a64a19 2019-12-21T16:04:30.311 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:714: run_osd_filestore: echo 'add osd2 d2b8f48d-5699-449e-8fc7-707402a64a19' 2019-12-21T16:04:30.312 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:715: run_osd_filestore: ceph-authtool --gen-print-key 2019-12-21T16:04:30.319 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:715: run_osd_filestore: OSD_SECRET=AQCOQv5dA63SEhAANGwzjoDXaU6w1AR8rfhUwg== 2019-12-21T16:04:30.319 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:716: run_osd_filestore: echo '{"cephx_secret": "AQCOQv5dA63SEhAANGwzjoDXaU6w1AR8rfhUwg=="}' 2019-12-21T16:04:30.319 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:717: run_osd_filestore: ceph osd new d2b8f48d-5699-449e-8fc7-707402a64a19 -i td/osd-scrub-repair/3/new.json 2019-12-21T16:04:30.664 INFO:tasks.workunit.client.0.smithi172.stdout:3 2019-12-21T16:04:30.674 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:718: run_osd_filestore: rm td/osd-scrub-repair/3/new.json 2019-12-21T16:04:30.675 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:719: run_osd_filestore: ceph-osd -i 3 --fsid=973e3fc1-e63e-4bde-9db6-84c68fd5e93e --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.119575/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCOQv5dA63SEhAANGwzjoDXaU6w1AR8rfhUwg== --osd-uuid d2b8f48d-5699-449e-8fc7-707402a64a19 --osd-objectstore=filestore 2019-12-21T16:04:30.696 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:04:30.693+0000 7f94c075aec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T16:04:30.714 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:04:30.711+0000 7f94c075aec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T16:04:30.716 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:04:30.713+0000 7f94c075aec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T16:04:31.007 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:04:31.005+0000 7f94c075aec0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2019-12-21T16:04:31.452 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:04:31.449+0000 7f94c075aec0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2019-12-21T16:04:31.452 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:04:31.449+0000 7f94c075aec0 -1 journal do_read_entry(4096): bad header magic 2019-12-21T16:04:31.452 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:04:31.449+0000 7f94c075aec0 -1 journal do_read_entry(4096): bad header magic 2019-12-21T16:04:31.959 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:721: run_osd_filestore: local key_fn=td/osd-scrub-repair/3/keyring 2019-12-21T16:04:31.960 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:722: run_osd_filestore: cat 2019-12-21T16:04:31.961 INFO:tasks.workunit.client.0.smithi172.stdout:adding osd3 key to auth repository 2019-12-21T16:04:31.961 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:726: run_osd_filestore: echo adding osd3 key to auth repository 2019-12-21T16:04:31.961 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:727: run_osd_filestore: ceph -i td/osd-scrub-repair/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2019-12-21T16:04:32.238 INFO:tasks.workunit.client.0.smithi172.stderr:Error EINVAL: key for osd.3 not found in provided keyring 2019-12-21T16:04:32.245 INFO:tasks.workunit.client.0.smithi172.stdout:start osd.3 2019-12-21T16:04:32.246 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:728: run_osd_filestore: echo start osd.3 2019-12-21T16:04:32.246 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:729: run_osd_filestore: ceph-osd -i 3 --fsid=973e3fc1-e63e-4bde-9db6-84c68fd5e93e --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.119575/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2019-12-21T16:04:32.247 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:732: run_osd_filestore: ceph osd dump --format=json 2019-12-21T16:04:32.247 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:732: run_osd_filestore: jq '.flags_set[]' 2019-12-21T16:04:32.247 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:732: run_osd_filestore: grep -q '"noup"' 2019-12-21T16:04:32.265 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:04:32.262+0000 7f28ccc22ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T16:04:32.266 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:04:32.264+0000 7f28ccc22ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T16:04:32.268 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:04:32.266+0000 7f28ccc22ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T16:04:32.269 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:04:32.267+0000 7f28ccc22ec0 -1 Falling back to public interface 2019-12-21T16:04:32.550 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:04:32.547+0000 7f28ccc22ec0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2019-12-21T16:04:32.550 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:04:32.547+0000 7f28ccc22ec0 -1 journal do_read_entry(8192): bad header magic 2019-12-21T16:04:32.551 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:04:32.547+0000 7f28ccc22ec0 -1 journal do_read_entry(8192): bad header magic 2019-12-21T16:04:32.557 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:04:32.554+0000 7f28ccc22ec0 -1 osd.3 0 log_to_monitors {default=true} 2019-12-21T16:04:32.627 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:735: run_osd_filestore: wait_for_osd up 3 2019-12-21T16:04:32.628 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:914: wait_for_osd: local state=up 2019-12-21T16:04:32.628 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local id=3 2019-12-21T16:04:32.628 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: status=1 2019-12-21T16:04:32.628 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i=0 )) 2019-12-21T16:04:32.628 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i < 300 )) 2019-12-21T16:04:32.628 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: echo 0 2019-12-21T16:04:32.628 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: ceph osd dump 2019-12-21T16:04:32.629 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: grep 'osd.3 up' 2019-12-21T16:04:32.629 INFO:tasks.workunit.client.0.smithi172.stdout:0 2019-12-21T16:04:32.877 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: sleep 1 2019-12-21T16:04:33.879 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i++ )) 2019-12-21T16:04:33.879 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i < 300 )) 2019-12-21T16:04:33.879 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: echo 1 2019-12-21T16:04:33.879 INFO:tasks.workunit.client.0.smithi172.stdout:1 2019-12-21T16:04:33.880 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: ceph osd dump 2019-12-21T16:04:33.880 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: grep 'osd.3 up' 2019-12-21T16:04:34.112 INFO:tasks.workunit.client.0.smithi172.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/154331,v1:127.0.0.1:6827/154331] [v2:127.0.0.1:6828/154331,v1:127.0.0.1:6829/154331] exists,up d2b8f48d-5699-449e-8fc7-707402a64a19 2019-12-21T16:04:34.112 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=0 2019-12-21T16:04:34.112 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: break 2019-12-21T16:04:34.112 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: return 0 2019-12-21T16:04:34.113 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:724: corrupt_and_repair_jerasure: create_rbd_pool 2019-12-21T16:04:34.113 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2019-12-21T16:04:34.335 INFO:tasks.workunit.client.0.smithi172.stderr:pool 'rbd' does not exist 2019-12-21T16:04:34.342 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:534: create_rbd_pool: create_pool rbd 4 2019-12-21T16:04:34.343 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:539: create_pool: ceph osd pool create rbd 4 2019-12-21T16:04:34.702 INFO:tasks.workunit.client.0.smithi172.stderr:pool 'rbd' created 2019-12-21T16:04:34.709 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: sleep 1 2019-12-21T16:04:35.711 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: rbd pool init rbd 2019-12-21T16:04:37.547 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:725: corrupt_and_repair_jerasure: wait_for_clean 2019-12-21T16:04:37.547 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1548: wait_for_clean: local cmd= 2019-12-21T16:04:37.547 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1549: wait_for_clean: local num_active_clean=-1 2019-12-21T16:04:37.547 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1550: wait_for_clean: local cur_active_clean 2019-12-21T16:04:37.547 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1551: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2019-12-21T16:04:37.548 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1551: wait_for_clean: get_timeout_delays 90 .1 2019-12-21T16:04:37.548 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1494: get_timeout_delays: shopt -q -o xtrace 2019-12-21T16:04:37.549 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1494: get_timeout_delays: echo true 2019-12-21T16:04:37.549 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1494: get_timeout_delays: local trace=true 2019-12-21T16:04:37.549 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1495: get_timeout_delays: true 2019-12-21T16:04:37.549 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1495: get_timeout_delays: shopt -u -o xtrace 2019-12-21T16:04:37.652 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1551: wait_for_clean: local -a delays 2019-12-21T16:04:37.652 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1552: wait_for_clean: local -i loop=0 2019-12-21T16:04:37.652 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1554: wait_for_clean: flush_pg_stats 2019-12-21T16:04:37.653 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1977: flush_pg_stats: local timeout=300 2019-12-21T16:04:37.653 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1979: flush_pg_stats: ceph osd ls 2019-12-21T16:04:37.883 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1979: flush_pg_stats: ids='0 2019-12-21T16:04:37.883 INFO:tasks.workunit.client.0.smithi172.stderr:1 2019-12-21T16:04:37.883 INFO:tasks.workunit.client.0.smithi172.stderr:2 2019-12-21T16:04:37.884 INFO:tasks.workunit.client.0.smithi172.stderr:3' 2019-12-21T16:04:37.884 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1980: flush_pg_stats: seqs= 2019-12-21T16:04:37.884 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: flush_pg_stats: for osd in $ids 2019-12-21T16:04:37.884 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2019-12-21T16:04:37.993 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: seq=21474836484 2019-12-21T16:04:37.993 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: flush_pg_stats: seqs=' 0-21474836484' 2019-12-21T16:04:37.993 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: flush_pg_stats: for osd in $ids 2019-12-21T16:04:37.993 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2019-12-21T16:04:38.101 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: seq=42949672964 2019-12-21T16:04:38.102 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: flush_pg_stats: seqs=' 0-21474836484 1-42949672964' 2019-12-21T16:04:38.102 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: flush_pg_stats: for osd in $ids 2019-12-21T16:04:38.102 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2019-12-21T16:04:38.210 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: seq=64424509443 2019-12-21T16:04:38.210 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: flush_pg_stats: seqs=' 0-21474836484 1-42949672964 2-64424509443' 2019-12-21T16:04:38.210 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: flush_pg_stats: for osd in $ids 2019-12-21T16:04:38.211 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2019-12-21T16:04:38.318 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: seq=85899345922 2019-12-21T16:04:38.318 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: flush_pg_stats: seqs=' 0-21474836484 1-42949672964 2-64424509443 3-85899345922' 2019-12-21T16:04:38.319 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: flush_pg_stats: for s in $seqs 2019-12-21T16:04:38.319 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: echo 0-21474836484 2019-12-21T16:04:38.319 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: cut -d - -f 1 2019-12-21T16:04:38.320 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: osd=0 2019-12-21T16:04:38.320 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: echo 0-21474836484 2019-12-21T16:04:38.320 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: cut -d - -f 2 2019-12-21T16:04:38.321 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: seq=21474836484 2019-12-21T16:04:38.321 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1989: flush_pg_stats: echo 'waiting osd.0 seq 21474836484' 2019-12-21T16:04:38.322 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 0 2019-12-21T16:04:38.322 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.0 seq 21474836484 2019-12-21T16:04:38.549 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 21474836483 -lt 21474836484 2019-12-21T16:04:38.549 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1991: flush_pg_stats: sleep 1 2019-12-21T16:04:39.550 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1992: flush_pg_stats: '[' 300 -eq 0 ']' 2019-12-21T16:04:39.550 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 0 2019-12-21T16:04:39.784 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 21474836485 -lt 21474836484 2019-12-21T16:04:39.784 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: flush_pg_stats: for s in $seqs 2019-12-21T16:04:39.784 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: echo 1-42949672964 2019-12-21T16:04:39.785 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: cut -d - -f 1 2019-12-21T16:04:39.785 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: osd=1 2019-12-21T16:04:39.786 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: echo 1-42949672964 2019-12-21T16:04:39.786 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: cut -d - -f 2 2019-12-21T16:04:39.786 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: seq=42949672964 2019-12-21T16:04:39.787 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1989: flush_pg_stats: echo 'waiting osd.1 seq 42949672964' 2019-12-21T16:04:39.787 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.1 seq 42949672964 2019-12-21T16:04:39.787 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 1 2019-12-21T16:04:40.014 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 42949672964 -lt 42949672964 2019-12-21T16:04:40.015 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: flush_pg_stats: for s in $seqs 2019-12-21T16:04:40.015 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: echo 2-64424509443 2019-12-21T16:04:40.015 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: cut -d - -f 1 2019-12-21T16:04:40.016 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: osd=2 2019-12-21T16:04:40.016 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: echo 2-64424509443 2019-12-21T16:04:40.016 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: cut -d - -f 2 2019-12-21T16:04:40.017 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: seq=64424509443 2019-12-21T16:04:40.017 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1989: flush_pg_stats: echo 'waiting osd.2 seq 64424509443' 2019-12-21T16:04:40.018 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 2 2019-12-21T16:04:40.018 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.2 seq 64424509443 2019-12-21T16:04:40.245 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 64424509443 -lt 64424509443 2019-12-21T16:04:40.245 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: flush_pg_stats: for s in $seqs 2019-12-21T16:04:40.245 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: echo 3-85899345922 2019-12-21T16:04:40.246 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: cut -d - -f 1 2019-12-21T16:04:40.246 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: osd=3 2019-12-21T16:04:40.246 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: echo 3-85899345922 2019-12-21T16:04:40.247 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: cut -d - -f 2 2019-12-21T16:04:40.247 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: seq=85899345922 2019-12-21T16:04:40.248 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1989: flush_pg_stats: echo 'waiting osd.3 seq 85899345922' 2019-12-21T16:04:40.248 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.3 seq 85899345922 2019-12-21T16:04:40.248 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 3 2019-12-21T16:04:40.475 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 85899345923 -lt 85899345922 2019-12-21T16:04:40.476 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1555: wait_for_clean: get_num_pgs 2019-12-21T16:04:40.476 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: ceph --format json status 2019-12-21T16:04:40.476 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: jq .pgmap.num_pgs 2019-12-21T16:04:40.745 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1555: wait_for_clean: test 4 == 0 2019-12-21T16:04:40.746 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1559: wait_for_clean: true 2019-12-21T16:04:40.746 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1563: wait_for_clean: get_num_active_clean 2019-12-21T16:04:40.746 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: get_num_active_clean: local expression 2019-12-21T16:04:40.746 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2019-12-21T16:04:40.746 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("stale") | not)' 2019-12-21T16:04:40.746 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: ceph --format json pg dump pgs 2019-12-21T16:04:40.746 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2019-12-21T16:04:40.970 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1563: wait_for_clean: cur_active_clean=4 2019-12-21T16:04:40.970 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1564: wait_for_clean: get_num_pgs 2019-12-21T16:04:40.971 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: ceph --format json status 2019-12-21T16:04:40.971 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: jq .pgmap.num_pgs 2019-12-21T16:04:41.254 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1564: wait_for_clean: test 4 = 4 2019-12-21T16:04:41.254 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1564: wait_for_clean: break 2019-12-21T16:04:41.255 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: return 0 2019-12-21T16:04:41.255 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:727: corrupt_and_repair_jerasure: create_ec_pool ecpool false k=2 m=2 2019-12-21T16:04:41.255 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: create_ec_pool: local pool_name=ecpool 2019-12-21T16:04:41.255 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2179: create_ec_pool: shift 2019-12-21T16:04:41.255 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2180: create_ec_pool: local allow_overwrites=false 2019-12-21T16:04:41.255 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: create_ec_pool: shift 2019-12-21T16:04:41.255 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=2 2019-12-21T16:04:41.627 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2019-12-21T16:04:41.627 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:539: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2019-12-21T16:04:42.163 INFO:tasks.workunit.client.0.smithi172.stderr:pool 'ecpool' created 2019-12-21T16:04:42.171 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: sleep 1 2019-12-21T16:04:43.173 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: create_ec_pool: '[' false = true ']' 2019-12-21T16:04:43.173 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2191: create_ec_pool: wait_for_clean 2019-12-21T16:04:43.173 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1548: wait_for_clean: local cmd= 2019-12-21T16:04:43.173 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1549: wait_for_clean: local num_active_clean=-1 2019-12-21T16:04:43.173 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1550: wait_for_clean: local cur_active_clean 2019-12-21T16:04:43.174 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1551: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2019-12-21T16:04:43.174 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1551: wait_for_clean: get_timeout_delays 90 .1 2019-12-21T16:04:43.174 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1494: get_timeout_delays: shopt -q -o xtrace 2019-12-21T16:04:43.175 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1494: get_timeout_delays: echo true 2019-12-21T16:04:43.175 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1494: get_timeout_delays: local trace=true 2019-12-21T16:04:43.175 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1495: get_timeout_delays: true 2019-12-21T16:04:43.175 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1495: get_timeout_delays: shopt -u -o xtrace 2019-12-21T16:04:43.278 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1551: wait_for_clean: local -a delays 2019-12-21T16:04:43.278 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1552: wait_for_clean: local -i loop=0 2019-12-21T16:04:43.279 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1554: wait_for_clean: flush_pg_stats 2019-12-21T16:04:43.279 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1977: flush_pg_stats: local timeout=300 2019-12-21T16:04:43.279 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1979: flush_pg_stats: ceph osd ls 2019-12-21T16:04:43.508 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1979: flush_pg_stats: ids='0 2019-12-21T16:04:43.508 INFO:tasks.workunit.client.0.smithi172.stderr:1 2019-12-21T16:04:43.509 INFO:tasks.workunit.client.0.smithi172.stderr:2 2019-12-21T16:04:43.509 INFO:tasks.workunit.client.0.smithi172.stderr:3' 2019-12-21T16:04:43.509 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1980: flush_pg_stats: seqs= 2019-12-21T16:04:43.509 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: flush_pg_stats: for osd in $ids 2019-12-21T16:04:43.510 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2019-12-21T16:04:43.617 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: seq=21474836487 2019-12-21T16:04:43.617 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: flush_pg_stats: seqs=' 0-21474836487' 2019-12-21T16:04:43.617 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: flush_pg_stats: for osd in $ids 2019-12-21T16:04:43.617 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2019-12-21T16:04:43.726 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: seq=42949672966 2019-12-21T16:04:43.726 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: flush_pg_stats: seqs=' 0-21474836487 1-42949672966' 2019-12-21T16:04:43.726 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: flush_pg_stats: for osd in $ids 2019-12-21T16:04:43.727 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2019-12-21T16:04:43.834 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: seq=64424509445 2019-12-21T16:04:43.834 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: flush_pg_stats: seqs=' 0-21474836487 1-42949672966 2-64424509445' 2019-12-21T16:04:43.834 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: flush_pg_stats: for osd in $ids 2019-12-21T16:04:43.835 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2019-12-21T16:04:43.944 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: seq=85899345925 2019-12-21T16:04:43.944 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: flush_pg_stats: seqs=' 0-21474836487 1-42949672966 2-64424509445 3-85899345925' 2019-12-21T16:04:43.944 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: flush_pg_stats: for s in $seqs 2019-12-21T16:04:43.945 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: echo 0-21474836487 2019-12-21T16:04:43.945 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: cut -d - -f 1 2019-12-21T16:04:43.945 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: osd=0 2019-12-21T16:04:43.946 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: echo 0-21474836487 2019-12-21T16:04:43.946 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: cut -d - -f 2 2019-12-21T16:04:43.947 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: seq=21474836487 2019-12-21T16:04:43.947 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1989: flush_pg_stats: echo 'waiting osd.0 seq 21474836487' 2019-12-21T16:04:43.947 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 0 2019-12-21T16:04:43.948 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.0 seq 21474836487 2019-12-21T16:04:44.238 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 21474836485 -lt 21474836487 2019-12-21T16:04:44.238 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1991: flush_pg_stats: sleep 1 2019-12-21T16:04:45.239 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1992: flush_pg_stats: '[' 300 -eq 0 ']' 2019-12-21T16:04:45.239 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 0 2019-12-21T16:04:45.475 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 21474836487 -lt 21474836487 2019-12-21T16:04:45.475 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: flush_pg_stats: for s in $seqs 2019-12-21T16:04:45.476 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: echo 1-42949672966 2019-12-21T16:04:45.476 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: cut -d - -f 1 2019-12-21T16:04:45.476 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: osd=1 2019-12-21T16:04:45.477 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: echo 1-42949672966 2019-12-21T16:04:45.477 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: cut -d - -f 2 2019-12-21T16:04:45.478 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: seq=42949672966 2019-12-21T16:04:45.478 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1989: flush_pg_stats: echo 'waiting osd.1 seq 42949672966' 2019-12-21T16:04:45.478 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.1 seq 42949672966 2019-12-21T16:04:45.479 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 1 2019-12-21T16:04:45.706 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 42949672966 -lt 42949672966 2019-12-21T16:04:45.706 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: flush_pg_stats: for s in $seqs 2019-12-21T16:04:45.707 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: echo 2-64424509445 2019-12-21T16:04:45.707 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: cut -d - -f 1 2019-12-21T16:04:45.708 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: osd=2 2019-12-21T16:04:45.712 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: echo 2-64424509445 2019-12-21T16:04:45.712 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: cut -d - -f 2 2019-12-21T16:04:45.712 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: seq=64424509445 2019-12-21T16:04:45.712 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1989: flush_pg_stats: echo 'waiting osd.2 seq 64424509445' 2019-12-21T16:04:45.712 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 2 2019-12-21T16:04:45.712 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.2 seq 64424509445 2019-12-21T16:04:45.937 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 64424509445 -lt 64424509445 2019-12-21T16:04:45.937 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: flush_pg_stats: for s in $seqs 2019-12-21T16:04:45.937 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: echo 3-85899345925 2019-12-21T16:04:45.938 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: cut -d - -f 1 2019-12-21T16:04:45.938 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: osd=3 2019-12-21T16:04:45.939 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: echo 3-85899345925 2019-12-21T16:04:45.939 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: cut -d - -f 2 2019-12-21T16:04:45.940 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: seq=85899345925 2019-12-21T16:04:45.940 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1989: flush_pg_stats: echo 'waiting osd.3 seq 85899345925' 2019-12-21T16:04:45.940 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 3 2019-12-21T16:04:45.940 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.3 seq 85899345925 2019-12-21T16:04:46.168 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 85899345925 -lt 85899345925 2019-12-21T16:04:46.168 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1555: wait_for_clean: get_num_pgs 2019-12-21T16:04:46.169 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: ceph --format json status 2019-12-21T16:04:46.169 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: jq .pgmap.num_pgs 2019-12-21T16:04:46.475 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1555: wait_for_clean: test 5 == 0 2019-12-21T16:04:46.475 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1559: wait_for_clean: true 2019-12-21T16:04:46.476 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1563: wait_for_clean: get_num_active_clean 2019-12-21T16:04:46.476 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: get_num_active_clean: local expression 2019-12-21T16:04:46.477 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2019-12-21T16:04:46.477 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("stale") | not)' 2019-12-21T16:04:46.477 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: ceph --format json pg dump pgs 2019-12-21T16:04:46.477 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2019-12-21T16:04:46.698 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1563: wait_for_clean: cur_active_clean=4 2019-12-21T16:04:46.698 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1564: wait_for_clean: get_num_pgs 2019-12-21T16:04:46.699 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: ceph --format json status 2019-12-21T16:04:46.699 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: jq .pgmap.num_pgs 2019-12-21T16:04:46.967 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1564: wait_for_clean: test 4 = 5 2019-12-21T16:04:46.967 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1565: wait_for_clean: test 4 '!=' -1 2019-12-21T16:04:46.968 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1566: wait_for_clean: loop=0 2019-12-21T16:04:46.968 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1567: wait_for_clean: num_active_clean=4 2019-12-21T16:04:46.968 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1575: wait_for_clean: eval 2019-12-21T16:04:46.968 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1576: wait_for_clean: sleep 0.1 2019-12-21T16:04:47.069 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1577: wait_for_clean: loop+=1 2019-12-21T16:04:47.069 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1559: wait_for_clean: true 2019-12-21T16:04:47.069 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1563: wait_for_clean: get_num_active_clean 2019-12-21T16:04:47.069 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: get_num_active_clean: local expression 2019-12-21T16:04:47.069 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2019-12-21T16:04:47.069 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("stale") | not)' 2019-12-21T16:04:47.070 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: ceph --format json pg dump pgs 2019-12-21T16:04:47.070 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2019-12-21T16:04:47.296 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1563: wait_for_clean: cur_active_clean=4 2019-12-21T16:04:47.297 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1564: wait_for_clean: get_num_pgs 2019-12-21T16:04:47.297 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: ceph --format json status 2019-12-21T16:04:47.297 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: jq .pgmap.num_pgs 2019-12-21T16:04:47.573 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1564: wait_for_clean: test 4 = 5 2019-12-21T16:04:47.573 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1565: wait_for_clean: test 4 '!=' 4 2019-12-21T16:04:47.574 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1568: wait_for_clean: get_is_making_recovery_progress 2019-12-21T16:04:47.574 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: get_is_making_recovery_progress: local recovery_progress 2019-12-21T16:04:47.574 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2019-12-21T16:04:47.574 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2019-12-21T16:04:47.574 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2019-12-21T16:04:47.574 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: ceph --format json status 2019-12-21T16:04:47.574 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2019-12-21T16:04:47.865 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: local progress=null 2019-12-21T16:04:47.865 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: test null '!=' null 2019-12-21T16:04:47.865 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1570: wait_for_clean: (( 1 >= 13 )) 2019-12-21T16:04:47.865 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1575: wait_for_clean: eval 2019-12-21T16:04:47.865 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1576: wait_for_clean: sleep 0.2 2019-12-21T16:04:48.066 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1577: wait_for_clean: loop+=1 2019-12-21T16:04:48.066 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1559: wait_for_clean: true 2019-12-21T16:04:48.067 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1563: wait_for_clean: get_num_active_clean 2019-12-21T16:04:48.067 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: get_num_active_clean: local expression 2019-12-21T16:04:48.067 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2019-12-21T16:04:48.067 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("stale") | not)' 2019-12-21T16:04:48.067 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: ceph --format json pg dump pgs 2019-12-21T16:04:48.067 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2019-12-21T16:04:48.290 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1563: wait_for_clean: cur_active_clean=4 2019-12-21T16:04:48.291 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1564: wait_for_clean: get_num_pgs 2019-12-21T16:04:48.291 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: ceph --format json status 2019-12-21T16:04:48.291 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: jq .pgmap.num_pgs 2019-12-21T16:04:48.560 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1564: wait_for_clean: test 4 = 5 2019-12-21T16:04:48.560 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1565: wait_for_clean: test 4 '!=' 4 2019-12-21T16:04:48.560 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1568: wait_for_clean: get_is_making_recovery_progress 2019-12-21T16:04:48.561 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: get_is_making_recovery_progress: local recovery_progress 2019-12-21T16:04:48.561 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2019-12-21T16:04:48.561 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2019-12-21T16:04:48.561 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2019-12-21T16:04:48.561 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: ceph --format json status 2019-12-21T16:04:48.561 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2019-12-21T16:04:48.829 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: local progress=null 2019-12-21T16:04:48.829 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: test null '!=' null 2019-12-21T16:04:48.830 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1570: wait_for_clean: (( 2 >= 13 )) 2019-12-21T16:04:48.830 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1575: wait_for_clean: eval 2019-12-21T16:04:48.830 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1576: wait_for_clean: sleep 0.4 2019-12-21T16:04:49.231 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1577: wait_for_clean: loop+=1 2019-12-21T16:04:49.232 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1559: wait_for_clean: true 2019-12-21T16:04:49.232 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1563: wait_for_clean: get_num_active_clean 2019-12-21T16:04:49.232 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: get_num_active_clean: local expression 2019-12-21T16:04:49.232 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2019-12-21T16:04:49.233 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("stale") | not)' 2019-12-21T16:04:49.233 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: ceph --format json pg dump pgs 2019-12-21T16:04:49.233 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2019-12-21T16:04:49.457 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1563: wait_for_clean: cur_active_clean=5 2019-12-21T16:04:49.457 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1564: wait_for_clean: get_num_pgs 2019-12-21T16:04:49.457 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: ceph --format json status 2019-12-21T16:04:49.457 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: jq .pgmap.num_pgs 2019-12-21T16:04:49.726 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1564: wait_for_clean: test 5 = 5 2019-12-21T16:04:49.726 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1564: wait_for_clean: break 2019-12-21T16:04:49.726 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: return 0 2019-12-21T16:04:49.727 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2192: create_ec_pool: return 0 2019-12-21T16:04:49.727 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:728: corrupt_and_repair_jerasure: corrupt_and_repair_erasure_coded td/osd-scrub-repair ecpool 2019-12-21T16:04:49.727 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: local dir=td/osd-scrub-repair 2019-12-21T16:04:49.727 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:255: corrupt_and_repair_erasure_coded: local poolname=ecpool 2019-12-21T16:04:49.727 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:257: corrupt_and_repair_erasure_coded: add_something td/osd-scrub-repair ecpool 2019-12-21T16:04:49.727 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:68: add_something: local dir=td/osd-scrub-repair 2019-12-21T16:04:49.728 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:69: add_something: local poolname=ecpool 2019-12-21T16:04:49.728 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:70: add_something: local obj=SOMETHING 2019-12-21T16:04:49.728 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local scrub=noscrub 2019-12-21T16:04:49.728 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: '[' noscrub = noscrub ']' 2019-12-21T16:04:49.728 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:75: add_something: ceph osd set noscrub 2019-12-21T16:04:50.165 INFO:tasks.workunit.client.0.smithi172.stderr:noscrub is set 2019-12-21T16:04:50.173 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: ceph osd set nodeep-scrub 2019-12-21T16:04:50.533 INFO:tasks.workunit.client.0.smithi172.stderr:nodeep-scrub is set 2019-12-21T16:04:50.541 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:82: add_something: local payload=ABCDEF 2019-12-21T16:04:50.541 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:83: add_something: echo ABCDEF 2019-12-21T16:04:50.541 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:84: add_something: rados --pool ecpool put SOMETHING td/osd-scrub-repair/ORIGINAL 2019-12-21T16:04:50.769 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:259: corrupt_and_repair_erasure_coded: get_primary ecpool SOMETHING 2019-12-21T16:04:50.769 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_primary: local poolname=ecpool 2019-12-21T16:04:50.769 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1133: get_primary: local objectname=SOMETHING 2019-12-21T16:04:50.769 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1135: get_primary: ceph --format json osd map ecpool SOMETHING 2019-12-21T16:04:50.770 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1136: get_primary: jq .acting_primary 2019-12-21T16:04:50.999 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:259: corrupt_and_repair_erasure_coded: local primary=3 2019-12-21T16:04:50.999 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:260: corrupt_and_repair_erasure_coded: osds=($(get_osds $poolname SOMETHING | sed -e "s/$primary//")) 2019-12-21T16:04:51.000 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:260: corrupt_and_repair_erasure_coded: get_osds ecpool SOMETHING 2019-12-21T16:04:51.000 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:957: get_osds: local poolname=ecpool 2019-12-21T16:04:51.000 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:958: get_osds: local objectname=SOMETHING 2019-12-21T16:04:51.000 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:260: corrupt_and_repair_erasure_coded: sed -e s/3// 2019-12-21T16:04:51.001 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: get_osds: ceph --format json osd map ecpool SOMETHING 2019-12-21T16:04:51.001 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: get_osds: jq '.acting | .[]' 2019-12-21T16:04:51.231 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: get_osds: local 'osds=3 2019-12-21T16:04:51.231 INFO:tasks.workunit.client.0.smithi172.stderr:1 2019-12-21T16:04:51.231 INFO:tasks.workunit.client.0.smithi172.stderr:2 2019-12-21T16:04:51.231 INFO:tasks.workunit.client.0.smithi172.stderr:0' 2019-12-21T16:04:51.232 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: echo 3 1 2 0 2019-12-21T16:04:51.234 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:260: corrupt_and_repair_erasure_coded: local -a osds 2019-12-21T16:04:51.234 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:261: corrupt_and_repair_erasure_coded: local not_primary_first=1 2019-12-21T16:04:51.234 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:262: corrupt_and_repair_erasure_coded: local not_primary_second=2 2019-12-21T16:04:51.234 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:265: corrupt_and_repair_erasure_coded: corrupt_and_repair_one td/osd-scrub-repair ecpool 3 2019-12-21T16:04:51.235 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:232: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2019-12-21T16:04:51.235 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:233: corrupt_and_repair_one: local poolname=ecpool 2019-12-21T16:04:51.235 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:234: corrupt_and_repair_one: local osd=3 2019-12-21T16:04:51.235 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:239: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2019-12-21T16:04:51.235 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: objectstore_tool: local dir=td/osd-scrub-repair 2019-12-21T16:04:51.235 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: objectstore_tool: shift 2019-12-21T16:04:51.235 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1235: objectstore_tool: local id=3 2019-12-21T16:04:51.236 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1236: objectstore_tool: shift 2019-12-21T16:04:51.236 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1238: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING remove 2019-12-21T16:04:51.236 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1205: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2019-12-21T16:04:51.236 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nowait: shift 2019-12-21T16:04:51.236 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1207: _objectstore_tool_nowait: local id=3 2019-12-21T16:04:51.236 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1208: _objectstore_tool_nowait: shift 2019-12-21T16:04:51.236 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1210: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2019-12-21T16:04:51.237 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2019-12-21T16:04:51.237 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2019-12-21T16:04:51.237 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2019-12-21T16:04:51.237 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2019-12-21T16:04:51.237 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2019-12-21T16:04:51.338 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2019-12-21T16:04:51.338 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1212: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING remove 2019-12-21T16:04:51.339 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1193: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2019-12-21T16:04:51.339 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1194: _objectstore_tool_nodown: shift 2019-12-21T16:04:51.339 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1195: _objectstore_tool_nodown: local id=3 2019-12-21T16:04:51.339 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: _objectstore_tool_nodown: shift 2019-12-21T16:04:51.339 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2019-12-21T16:04:51.339 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING remove 2019-12-21T16:04:52.148 INFO:tasks.workunit.client.0.smithi172.stdout:remove 0#2:eb822e21:::SOMETHING:head# 2019-12-21T16:04:52.657 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1213: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2019-12-21T16:04:52.657 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:842: activate_osd: local dir=td/osd-scrub-repair 2019-12-21T16:04:52.657 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: shift 2019-12-21T16:04:52.658 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: local id=3 2019-12-21T16:04:52.658 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: shift 2019-12-21T16:04:52.658 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local osd_data=td/osd-scrub-repair/3 2019-12-21T16:04:52.658 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local 'ceph_args=--fsid=973e3fc1-e63e-4bde-9db6-84c68fd5e93e --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2019-12-21T16:04:52.658 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2019-12-21T16:04:52.658 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-journal-size=100' 2019-12-21T16:04:52.658 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2019-12-21T16:04:52.659 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2019-12-21T16:04:52.659 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2019-12-21T16:04:52.659 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --chdir=' 2019-12-21T16:04:52.659 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+= 2019-12-21T16:04:52.659 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2019-12-21T16:04:52.659 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: get_asok_path 2019-12-21T16:04:52.659 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2019-12-21T16:04:52.660 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2019-12-21T16:04:52.660 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2019-12-21T16:04:52.660 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T16:04:52.660 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.119575 2019-12-21T16:04:52.660 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.119575/$cluster-$name.asok' 2019-12-21T16:04:52.660 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.119575/$cluster-$name.asok' 2019-12-21T16:04:52.660 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --debug-osd=20' 2019-12-21T16:04:52.661 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2019-12-21T16:04:52.661 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2019-12-21T16:04:52.661 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2019-12-21T16:04:52.661 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2019-12-21T16:04:52.661 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2019-12-21T16:04:52.661 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' ' 2019-12-21T16:04:52.661 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+= 2019-12-21T16:04:52.661 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: mkdir -p td/osd-scrub-repair/3 2019-12-21T16:04:52.662 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: echo start osd.3 2019-12-21T16:04:52.662 INFO:tasks.workunit.client.0.smithi172.stderr:start osd.3 2019-12-21T16:04:52.663 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph-osd -i 3 --fsid=973e3fc1-e63e-4bde-9db6-84c68fd5e93e --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.119575/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2019-12-21T16:04:52.663 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: cat td/osd-scrub-repair/3/whoami 2019-12-21T16:04:52.663 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: '[' 3 = 3 ']' 2019-12-21T16:04:52.663 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph osd dump --format=json 2019-12-21T16:04:52.663 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: jq '.flags_set[]' 2019-12-21T16:04:52.663 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: grep -q '"noup"' 2019-12-21T16:04:52.678 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:04:52.676+0000 7f9748ba3ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T16:04:52.680 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:04:52.679+0000 7f9748ba3ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T16:04:52.682 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:04:52.681+0000 7f9748ba3ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T16:04:52.683 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:04:52.682+0000 7f9748ba3ec0 -1 Falling back to public interface 2019-12-21T16:04:52.895 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: wait_for_osd up 3 2019-12-21T16:04:52.895 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:914: wait_for_osd: local state=up 2019-12-21T16:04:52.895 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local id=3 2019-12-21T16:04:52.896 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: status=1 2019-12-21T16:04:52.896 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i=0 )) 2019-12-21T16:04:52.896 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i < 300 )) 2019-12-21T16:04:52.896 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: echo 0 2019-12-21T16:04:52.896 INFO:tasks.workunit.client.0.smithi172.stderr:0 2019-12-21T16:04:52.896 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: ceph osd dump 2019-12-21T16:04:52.897 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: grep 'osd.3 up' 2019-12-21T16:04:53.136 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:04:53.135+0000 7f9748ba3ec0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2019-12-21T16:04:53.144 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:04:53.143+0000 7f9748ba3ec0 -1 journal do_read_entry(286720): bad header magic 2019-12-21T16:04:53.144 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:04:53.143+0000 7f9748ba3ec0 -1 journal do_read_entry(286720): bad header magic 2019-12-21T16:04:53.152 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:04:53.151+0000 7f9748ba3ec0 -1 osd.3 36 log_to_monitors {default=true} 2019-12-21T16:04:53.162 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: sleep 1 2019-12-21T16:04:54.164 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i++ )) 2019-12-21T16:04:54.164 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i < 300 )) 2019-12-21T16:04:54.164 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: echo 1 2019-12-21T16:04:54.164 INFO:tasks.workunit.client.0.smithi172.stderr:1 2019-12-21T16:04:54.164 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: ceph osd dump 2019-12-21T16:04:54.164 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: grep 'osd.3 up' 2019-12-21T16:04:54.394 INFO:tasks.workunit.client.0.smithi172.stderr:osd.3 up in weight 1 up_from 40 up_thru 40 down_at 37 last_clean_interval [20,36) [v2:127.0.0.1:6826/155955,v1:127.0.0.1:6827/155955] [v2:127.0.0.1:6828/155955,v1:127.0.0.1:6829/155955] exists,up d2b8f48d-5699-449e-8fc7-707402a64a19 2019-12-21T16:04:54.394 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=0 2019-12-21T16:04:54.394 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: break 2019-12-21T16:04:54.394 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: return 0 2019-12-21T16:04:54.394 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1239: objectstore_tool: wait_for_clean 2019-12-21T16:04:54.394 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1548: wait_for_clean: local cmd= 2019-12-21T16:04:54.395 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1549: wait_for_clean: local num_active_clean=-1 2019-12-21T16:04:54.395 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1550: wait_for_clean: local cur_active_clean 2019-12-21T16:04:54.395 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1551: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2019-12-21T16:04:54.395 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1551: wait_for_clean: get_timeout_delays 90 .1 2019-12-21T16:04:54.395 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1494: get_timeout_delays: shopt -q -o xtrace 2019-12-21T16:04:54.395 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1494: get_timeout_delays: echo true 2019-12-21T16:04:54.395 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1494: get_timeout_delays: local trace=true 2019-12-21T16:04:54.395 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1495: get_timeout_delays: true 2019-12-21T16:04:54.396 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1495: get_timeout_delays: shopt -u -o xtrace 2019-12-21T16:04:54.487 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1551: wait_for_clean: local -a delays 2019-12-21T16:04:54.487 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1552: wait_for_clean: local -i loop=0 2019-12-21T16:04:54.487 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1554: wait_for_clean: flush_pg_stats 2019-12-21T16:04:54.487 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1977: flush_pg_stats: local timeout=300 2019-12-21T16:04:54.488 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1979: flush_pg_stats: ceph osd ls 2019-12-21T16:04:54.713 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1979: flush_pg_stats: ids='0 2019-12-21T16:04:54.714 INFO:tasks.workunit.client.0.smithi172.stderr:1 2019-12-21T16:04:54.714 INFO:tasks.workunit.client.0.smithi172.stderr:2 2019-12-21T16:04:54.714 INFO:tasks.workunit.client.0.smithi172.stderr:3' 2019-12-21T16:04:54.714 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1980: flush_pg_stats: seqs= 2019-12-21T16:04:54.714 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: flush_pg_stats: for osd in $ids 2019-12-21T16:04:54.714 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2019-12-21T16:04:54.822 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: seq=21474836490 2019-12-21T16:04:54.822 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: flush_pg_stats: seqs=' 0-21474836490' 2019-12-21T16:04:54.823 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: flush_pg_stats: for osd in $ids 2019-12-21T16:04:54.823 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2019-12-21T16:04:54.931 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: seq=42949672969 2019-12-21T16:04:54.931 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: flush_pg_stats: seqs=' 0-21474836490 1-42949672969' 2019-12-21T16:04:54.931 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: flush_pg_stats: for osd in $ids 2019-12-21T16:04:54.932 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2019-12-21T16:04:55.041 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: seq=64424509448 2019-12-21T16:04:55.041 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: flush_pg_stats: seqs=' 0-21474836490 1-42949672969 2-64424509448' 2019-12-21T16:04:55.041 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: flush_pg_stats: for osd in $ids 2019-12-21T16:04:55.041 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2019-12-21T16:04:55.150 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: seq=171798691842 2019-12-21T16:04:55.150 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: flush_pg_stats: seqs=' 0-21474836490 1-42949672969 2-64424509448 3-171798691842' 2019-12-21T16:04:55.150 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: flush_pg_stats: for s in $seqs 2019-12-21T16:04:55.151 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: echo 0-21474836490 2019-12-21T16:04:55.151 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: cut -d - -f 1 2019-12-21T16:04:55.151 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: osd=0 2019-12-21T16:04:55.152 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: echo 0-21474836490 2019-12-21T16:04:55.152 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: cut -d - -f 2 2019-12-21T16:04:55.153 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: seq=21474836490 2019-12-21T16:04:55.153 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1989: flush_pg_stats: echo 'waiting osd.0 seq 21474836490' 2019-12-21T16:04:55.153 INFO:tasks.workunit.client.0.smithi172.stderr:waiting osd.0 seq 21474836490 2019-12-21T16:04:55.154 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 0 2019-12-21T16:04:55.381 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 21474836490 -lt 21474836490 2019-12-21T16:04:55.381 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: flush_pg_stats: for s in $seqs 2019-12-21T16:04:55.381 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: echo 1-42949672969 2019-12-21T16:04:55.381 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: cut -d - -f 1 2019-12-21T16:04:55.382 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: osd=1 2019-12-21T16:04:55.383 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: echo 1-42949672969 2019-12-21T16:04:55.383 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: cut -d - -f 2 2019-12-21T16:04:55.383 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: seq=42949672969 2019-12-21T16:04:55.384 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1989: flush_pg_stats: echo 'waiting osd.1 seq 42949672969' 2019-12-21T16:04:55.384 INFO:tasks.workunit.client.0.smithi172.stderr:waiting osd.1 seq 42949672969 2019-12-21T16:04:55.384 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 1 2019-12-21T16:04:55.611 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 42949672969 -lt 42949672969 2019-12-21T16:04:55.611 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: flush_pg_stats: for s in $seqs 2019-12-21T16:04:55.612 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: echo 2-64424509448 2019-12-21T16:04:55.612 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: cut -d - -f 1 2019-12-21T16:04:55.612 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: osd=2 2019-12-21T16:04:55.613 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: echo 2-64424509448 2019-12-21T16:04:55.613 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: cut -d - -f 2 2019-12-21T16:04:55.614 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: seq=64424509448 2019-12-21T16:04:55.614 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1989: flush_pg_stats: echo 'waiting osd.2 seq 64424509448' 2019-12-21T16:04:55.614 INFO:tasks.workunit.client.0.smithi172.stderr:waiting osd.2 seq 64424509448 2019-12-21T16:04:55.614 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 2 2019-12-21T16:04:55.841 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 64424509447 -lt 64424509448 2019-12-21T16:04:55.841 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1991: flush_pg_stats: sleep 1 2019-12-21T16:04:56.842 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1992: flush_pg_stats: '[' 300 -eq 0 ']' 2019-12-21T16:04:56.843 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 2 2019-12-21T16:04:57.080 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 64424509447 -lt 64424509448 2019-12-21T16:04:57.080 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1991: flush_pg_stats: sleep 1 2019-12-21T16:04:58.081 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1992: flush_pg_stats: '[' 299 -eq 0 ']' 2019-12-21T16:04:58.082 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 2 2019-12-21T16:04:58.319 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 64424509449 -lt 64424509448 2019-12-21T16:04:58.319 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: flush_pg_stats: for s in $seqs 2019-12-21T16:04:58.319 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: echo 3-171798691842 2019-12-21T16:04:58.320 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: cut -d - -f 1 2019-12-21T16:04:58.320 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: osd=3 2019-12-21T16:04:58.321 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: echo 3-171798691842 2019-12-21T16:04:58.321 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: cut -d - -f 2 2019-12-21T16:04:58.321 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: seq=171798691842 2019-12-21T16:04:58.322 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1989: flush_pg_stats: echo 'waiting osd.3 seq 171798691842' 2019-12-21T16:04:58.322 INFO:tasks.workunit.client.0.smithi172.stderr:waiting osd.3 seq 171798691842 2019-12-21T16:04:58.322 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 3 2019-12-21T16:04:58.547 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 171798691842 -lt 171798691842 2019-12-21T16:04:58.549 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1555: wait_for_clean: get_num_pgs 2019-12-21T16:04:58.549 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: ceph --format json status 2019-12-21T16:04:58.549 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: jq .pgmap.num_pgs 2019-12-21T16:04:58.816 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1555: wait_for_clean: test 5 == 0 2019-12-21T16:04:58.816 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1559: wait_for_clean: true 2019-12-21T16:04:58.817 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1563: wait_for_clean: get_num_active_clean 2019-12-21T16:04:58.817 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: get_num_active_clean: local expression 2019-12-21T16:04:58.817 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2019-12-21T16:04:58.817 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("stale") | not)' 2019-12-21T16:04:58.818 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: ceph --format json pg dump pgs 2019-12-21T16:04:58.818 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2019-12-21T16:04:59.039 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1563: wait_for_clean: cur_active_clean=5 2019-12-21T16:04:59.040 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1564: wait_for_clean: get_num_pgs 2019-12-21T16:04:59.040 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: ceph --format json status 2019-12-21T16:04:59.040 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: jq .pgmap.num_pgs 2019-12-21T16:04:59.328 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1564: wait_for_clean: test 5 = 5 2019-12-21T16:04:59.328 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1564: wait_for_clean: break 2019-12-21T16:04:59.328 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: return 0 2019-12-21T16:04:59.329 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:243: corrupt_and_repair_one: get_pg ecpool SOMETHING 2019-12-21T16:04:59.329 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1029: get_pg: local poolname=ecpool 2019-12-21T16:04:59.329 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1030: get_pg: local objectname=SOMETHING 2019-12-21T16:04:59.329 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1032: get_pg: ceph --format json osd map ecpool SOMETHING 2019-12-21T16:04:59.329 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1032: get_pg: jq -r .pgid 2019-12-21T16:04:59.560 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:243: corrupt_and_repair_one: local pg=2.0 2019-12-21T16:04:59.560 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:244: corrupt_and_repair_one: repair 2.0 2019-12-21T16:04:59.560 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: repair: local pgid=2.0 2019-12-21T16:04:59.561 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1665: repair: get_last_scrub_stamp 2.0 2019-12-21T16:04:59.561 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=2.0 2019-12-21T16:04:59.561 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T16:04:59.561 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T16:04:59.561 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2019-12-21T16:04:59.785 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1665: repair: local last_scrub=2019-12-21T16:04:42.082821+0000 2019-12-21T16:04:59.785 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1666: repair: ceph pg repair 2.0 2019-12-21T16:04:59.998 INFO:tasks.workunit.client.0.smithi172.stderr:instructing pg 2.0s0 on osd.3 to repair 2019-12-21T16:05:00.007 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: repair: wait_for_scrub 2.0 2019-12-21T16:04:42.082821+0000 2019-12-21T16:05:00.007 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1789: wait_for_scrub: local pgid=2.0 2019-12-21T16:05:00.007 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1790: wait_for_scrub: local last_scrub=2019-12-21T16:04:42.082821+0000 2019-12-21T16:05:00.007 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1791: wait_for_scrub: local sname=last_scrub_stamp 2019-12-21T16:05:00.007 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i=0 )) 2019-12-21T16:05:00.007 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T16:05:00.008 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2019-12-21T16:05:00.008 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=2.0 2019-12-21T16:05:00.008 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T16:05:00.008 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T16:05:00.008 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2019-12-21T16:05:00.229 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T16:04:42.082821+0000 '>' 2019-12-21T16:04:42.082821+0000 2019-12-21T16:05:00.229 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T16:05:01.230 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T16:05:01.230 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T16:05:01.231 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2019-12-21T16:05:01.231 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=2.0 2019-12-21T16:05:01.231 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T16:05:01.231 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T16:05:01.232 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2019-12-21T16:05:01.463 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T16:04:42.082821+0000 '>' 2019-12-21T16:04:42.082821+0000 2019-12-21T16:05:01.463 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T16:05:02.464 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T16:05:02.464 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T16:05:02.465 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2019-12-21T16:05:02.465 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=2.0 2019-12-21T16:05:02.465 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T16:05:02.465 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T16:05:02.465 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2019-12-21T16:05:02.698 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T16:04:42.082821+0000 '>' 2019-12-21T16:04:42.082821+0000 2019-12-21T16:05:02.699 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T16:05:03.699 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T16:05:03.700 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T16:05:03.700 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2019-12-21T16:05:03.701 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=2.0 2019-12-21T16:05:03.701 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T16:05:03.701 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T16:05:03.701 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2019-12-21T16:05:03.927 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T16:04:42.082821+0000 '>' 2019-12-21T16:04:42.082821+0000 2019-12-21T16:05:03.928 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T16:05:04.929 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T16:05:04.929 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T16:05:04.930 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2019-12-21T16:05:04.930 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=2.0 2019-12-21T16:05:04.930 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T16:05:04.930 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T16:05:04.930 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2019-12-21T16:05:05.167 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T16:04:42.082821+0000 '>' 2019-12-21T16:04:42.082821+0000 2019-12-21T16:05:05.167 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T16:05:06.169 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T16:05:06.169 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T16:05:06.169 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2019-12-21T16:05:06.169 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=2.0 2019-12-21T16:05:06.169 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T16:05:06.170 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T16:05:06.170 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2019-12-21T16:05:06.406 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T16:04:42.082821+0000 '>' 2019-12-21T16:04:42.082821+0000 2019-12-21T16:05:06.406 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T16:05:07.407 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T16:05:07.407 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T16:05:07.408 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2019-12-21T16:05:07.408 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=2.0 2019-12-21T16:05:07.409 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T16:05:07.409 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T16:05:07.409 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2019-12-21T16:05:07.639 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T16:04:42.082821+0000 '>' 2019-12-21T16:04:42.082821+0000 2019-12-21T16:05:07.640 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T16:05:08.640 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T16:05:08.640 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T16:05:08.641 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2019-12-21T16:05:08.641 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=2.0 2019-12-21T16:05:08.642 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T16:05:08.642 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T16:05:08.642 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2019-12-21T16:05:08.868 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T16:04:42.082821+0000 '>' 2019-12-21T16:04:42.082821+0000 2019-12-21T16:05:08.868 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T16:05:09.869 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T16:05:09.869 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T16:05:09.870 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2019-12-21T16:05:09.870 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=2.0 2019-12-21T16:05:09.870 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T16:05:09.871 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T16:05:09.871 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2019-12-21T16:05:10.101 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T16:05:06.199497+0000 '>' 2019-12-21T16:04:42.082821+0000 2019-12-21T16:05:10.101 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1795: wait_for_scrub: return 0 2019-12-21T16:05:10.101 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:248: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2019-12-21T16:05:10.101 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: objectstore_tool: local dir=td/osd-scrub-repair 2019-12-21T16:05:10.101 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: objectstore_tool: shift 2019-12-21T16:05:10.102 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1235: objectstore_tool: local id=3 2019-12-21T16:05:10.102 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1236: objectstore_tool: shift 2019-12-21T16:05:10.102 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1238: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING list-attrs 2019-12-21T16:05:10.102 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1205: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2019-12-21T16:05:10.102 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nowait: shift 2019-12-21T16:05:10.102 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1207: _objectstore_tool_nowait: local id=3 2019-12-21T16:05:10.102 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1208: _objectstore_tool_nowait: shift 2019-12-21T16:05:10.103 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1210: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2019-12-21T16:05:10.103 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2019-12-21T16:05:10.103 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2019-12-21T16:05:10.103 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2019-12-21T16:05:10.103 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2019-12-21T16:05:10.103 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2019-12-21T16:05:10.206 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2019-12-21T16:05:10.206 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1212: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING list-attrs 2019-12-21T16:05:10.206 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1193: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2019-12-21T16:05:10.207 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1194: _objectstore_tool_nodown: shift 2019-12-21T16:05:10.207 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1195: _objectstore_tool_nodown: local id=3 2019-12-21T16:05:10.207 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: _objectstore_tool_nodown: shift 2019-12-21T16:05:10.207 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2019-12-21T16:05:10.207 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING list-attrs 2019-12-21T16:05:10.693 INFO:tasks.workunit.client.0.smithi172.stderr:Error getting attr on : 2.0s0_head,0#-4:00000000:::scrub_2.0s0:head#, (61) No data available 2019-12-21T16:05:10.693 INFO:tasks.workunit.client.0.smithi172.stdout:_ 2019-12-21T16:05:10.693 INFO:tasks.workunit.client.0.smithi172.stdout:hinfo_key 2019-12-21T16:05:10.694 INFO:tasks.workunit.client.0.smithi172.stdout:snapset 2019-12-21T16:05:10.903 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1213: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2019-12-21T16:05:10.903 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:842: activate_osd: local dir=td/osd-scrub-repair 2019-12-21T16:05:10.903 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: shift 2019-12-21T16:05:10.903 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: local id=3 2019-12-21T16:05:10.904 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: shift 2019-12-21T16:05:10.904 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local osd_data=td/osd-scrub-repair/3 2019-12-21T16:05:10.904 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local 'ceph_args=--fsid=973e3fc1-e63e-4bde-9db6-84c68fd5e93e --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2019-12-21T16:05:10.904 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2019-12-21T16:05:10.904 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-journal-size=100' 2019-12-21T16:05:10.904 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2019-12-21T16:05:10.904 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2019-12-21T16:05:10.905 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2019-12-21T16:05:10.905 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --chdir=' 2019-12-21T16:05:10.905 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+= 2019-12-21T16:05:10.905 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2019-12-21T16:05:10.905 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: get_asok_path 2019-12-21T16:05:10.905 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2019-12-21T16:05:10.906 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2019-12-21T16:05:10.906 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2019-12-21T16:05:10.906 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T16:05:10.906 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.119575 2019-12-21T16:05:10.906 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.119575/$cluster-$name.asok' 2019-12-21T16:05:10.906 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.119575/$cluster-$name.asok' 2019-12-21T16:05:10.906 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --debug-osd=20' 2019-12-21T16:05:10.907 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2019-12-21T16:05:10.907 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2019-12-21T16:05:10.907 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2019-12-21T16:05:10.907 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2019-12-21T16:05:10.907 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2019-12-21T16:05:10.907 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' ' 2019-12-21T16:05:10.907 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+= 2019-12-21T16:05:10.908 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: mkdir -p td/osd-scrub-repair/3 2019-12-21T16:05:10.908 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: echo start osd.3 2019-12-21T16:05:10.908 INFO:tasks.workunit.client.0.smithi172.stderr:start osd.3 2019-12-21T16:05:10.908 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph-osd -i 3 --fsid=973e3fc1-e63e-4bde-9db6-84c68fd5e93e --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.119575/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2019-12-21T16:05:10.915 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: cat td/osd-scrub-repair/3/whoami 2019-12-21T16:05:10.915 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: '[' 3 = 3 ']' 2019-12-21T16:05:10.916 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph osd dump --format=json 2019-12-21T16:05:10.916 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: jq '.flags_set[]' 2019-12-21T16:05:10.916 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: grep -q '"noup"' 2019-12-21T16:05:10.921 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:05:10.919+0000 7f5e9aa76ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T16:05:10.924 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:05:10.921+0000 7f5e9aa76ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T16:05:10.926 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:05:10.923+0000 7f5e9aa76ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T16:05:10.927 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:05:10.924+0000 7f5e9aa76ec0 -1 Falling back to public interface 2019-12-21T16:05:11.195 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: wait_for_osd up 3 2019-12-21T16:05:11.195 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:914: wait_for_osd: local state=up 2019-12-21T16:05:11.195 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local id=3 2019-12-21T16:05:11.195 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: status=1 2019-12-21T16:05:11.195 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i=0 )) 2019-12-21T16:05:11.195 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i < 300 )) 2019-12-21T16:05:11.195 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: echo 0 2019-12-21T16:05:11.196 INFO:tasks.workunit.client.0.smithi172.stderr:0 2019-12-21T16:05:11.196 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: ceph osd dump 2019-12-21T16:05:11.196 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: grep 'osd.3 up' 2019-12-21T16:05:11.335 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:05:11.333+0000 7f5e9aa76ec0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2019-12-21T16:05:11.353 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:05:11.341+0000 7f5e9aa76ec0 -1 journal do_read_entry(434176): bad header magic 2019-12-21T16:05:11.353 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:05:11.341+0000 7f5e9aa76ec0 -1 journal do_read_entry(434176): bad header magic 2019-12-21T16:05:11.353 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:05:11.350+0000 7f5e9aa76ec0 -1 osd.3 42 log_to_monitors {default=true} 2019-12-21T16:05:11.458 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: sleep 1 2019-12-21T16:05:12.459 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i++ )) 2019-12-21T16:05:12.460 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i < 300 )) 2019-12-21T16:05:12.460 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: echo 1 2019-12-21T16:05:12.460 INFO:tasks.workunit.client.0.smithi172.stderr:1 2019-12-21T16:05:12.460 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: ceph osd dump 2019-12-21T16:05:12.460 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: grep 'osd.3 up' 2019-12-21T16:05:12.728 INFO:tasks.workunit.client.0.smithi172.stderr:osd.3 up in weight 1 up_from 46 up_thru 46 down_at 43 last_clean_interval [40,42) [v2:127.0.0.1:6826/157002,v1:127.0.0.1:6827/157002] [v2:127.0.0.1:6828/157002,v1:127.0.0.1:6829/157002] exists,up d2b8f48d-5699-449e-8fc7-707402a64a19 2019-12-21T16:05:12.728 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=0 2019-12-21T16:05:12.728 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: break 2019-12-21T16:05:12.728 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: return 0 2019-12-21T16:05:12.729 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1239: objectstore_tool: wait_for_clean 2019-12-21T16:05:12.729 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1548: wait_for_clean: local cmd= 2019-12-21T16:05:12.729 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1549: wait_for_clean: local num_active_clean=-1 2019-12-21T16:05:12.729 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1550: wait_for_clean: local cur_active_clean 2019-12-21T16:05:12.729 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1551: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2019-12-21T16:05:12.729 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1551: wait_for_clean: get_timeout_delays 90 .1 2019-12-21T16:05:12.730 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1494: get_timeout_delays: shopt -q -o xtrace 2019-12-21T16:05:12.730 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1494: get_timeout_delays: echo true 2019-12-21T16:05:12.730 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1494: get_timeout_delays: local trace=true 2019-12-21T16:05:12.730 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1495: get_timeout_delays: true 2019-12-21T16:05:12.730 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1495: get_timeout_delays: shopt -u -o xtrace 2019-12-21T16:05:12.821 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1551: wait_for_clean: local -a delays 2019-12-21T16:05:12.822 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1552: wait_for_clean: local -i loop=0 2019-12-21T16:05:12.822 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1554: wait_for_clean: flush_pg_stats 2019-12-21T16:05:12.822 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1977: flush_pg_stats: local timeout=300 2019-12-21T16:05:12.822 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1979: flush_pg_stats: ceph osd ls 2019-12-21T16:05:13.051 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1979: flush_pg_stats: ids='0 2019-12-21T16:05:13.051 INFO:tasks.workunit.client.0.smithi172.stderr:1 2019-12-21T16:05:13.051 INFO:tasks.workunit.client.0.smithi172.stderr:2 2019-12-21T16:05:13.052 INFO:tasks.workunit.client.0.smithi172.stderr:3' 2019-12-21T16:05:13.052 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1980: flush_pg_stats: seqs= 2019-12-21T16:05:13.052 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: flush_pg_stats: for osd in $ids 2019-12-21T16:05:13.052 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2019-12-21T16:05:13.160 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: seq=21474836494 2019-12-21T16:05:13.161 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: flush_pg_stats: seqs=' 0-21474836494' 2019-12-21T16:05:13.161 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: flush_pg_stats: for osd in $ids 2019-12-21T16:05:13.161 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2019-12-21T16:05:13.269 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: seq=42949672974 2019-12-21T16:05:13.269 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: flush_pg_stats: seqs=' 0-21474836494 1-42949672974' 2019-12-21T16:05:13.270 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: flush_pg_stats: for osd in $ids 2019-12-21T16:05:13.270 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2019-12-21T16:05:13.378 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: seq=64424509453 2019-12-21T16:05:13.378 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: flush_pg_stats: seqs=' 0-21474836494 1-42949672974 2-64424509453' 2019-12-21T16:05:13.378 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: flush_pg_stats: for osd in $ids 2019-12-21T16:05:13.378 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2019-12-21T16:05:13.493 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: seq=197568495618 2019-12-21T16:05:13.493 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: flush_pg_stats: seqs=' 0-21474836494 1-42949672974 2-64424509453 3-197568495618' 2019-12-21T16:05:13.493 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: flush_pg_stats: for s in $seqs 2019-12-21T16:05:13.494 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: echo 0-21474836494 2019-12-21T16:05:13.494 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: cut -d - -f 1 2019-12-21T16:05:13.494 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: osd=0 2019-12-21T16:05:13.494 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: echo 0-21474836494 2019-12-21T16:05:13.494 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: cut -d - -f 2 2019-12-21T16:05:13.494 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: seq=21474836494 2019-12-21T16:05:13.495 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1989: flush_pg_stats: echo 'waiting osd.0 seq 21474836494' 2019-12-21T16:05:13.495 INFO:tasks.workunit.client.0.smithi172.stderr:waiting osd.0 seq 21474836494 2019-12-21T16:05:13.495 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 0 2019-12-21T16:05:13.715 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 21474836493 -lt 21474836494 2019-12-21T16:05:13.715 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1991: flush_pg_stats: sleep 1 2019-12-21T16:05:14.716 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1992: flush_pg_stats: '[' 300 -eq 0 ']' 2019-12-21T16:05:14.716 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 0 2019-12-21T16:05:14.945 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 21474836493 -lt 21474836494 2019-12-21T16:05:14.945 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1991: flush_pg_stats: sleep 1 2019-12-21T16:05:15.946 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1992: flush_pg_stats: '[' 299 -eq 0 ']' 2019-12-21T16:05:15.947 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 0 2019-12-21T16:05:16.181 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 21474836495 -lt 21474836494 2019-12-21T16:05:16.182 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: flush_pg_stats: for s in $seqs 2019-12-21T16:05:16.182 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: echo 1-42949672974 2019-12-21T16:05:16.182 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: cut -d - -f 1 2019-12-21T16:05:16.184 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: osd=1 2019-12-21T16:05:16.184 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: echo 1-42949672974 2019-12-21T16:05:16.184 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: cut -d - -f 2 2019-12-21T16:05:16.184 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: seq=42949672974 2019-12-21T16:05:16.185 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1989: flush_pg_stats: echo 'waiting osd.1 seq 42949672974' 2019-12-21T16:05:16.185 INFO:tasks.workunit.client.0.smithi172.stderr:waiting osd.1 seq 42949672974 2019-12-21T16:05:16.185 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 1 2019-12-21T16:05:16.411 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 42949672974 -lt 42949672974 2019-12-21T16:05:16.411 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: flush_pg_stats: for s in $seqs 2019-12-21T16:05:16.412 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: echo 2-64424509453 2019-12-21T16:05:16.412 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: cut -d - -f 1 2019-12-21T16:05:16.412 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: osd=2 2019-12-21T16:05:16.413 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: echo 2-64424509453 2019-12-21T16:05:16.413 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: cut -d - -f 2 2019-12-21T16:05:16.415 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: seq=64424509453 2019-12-21T16:05:16.415 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1989: flush_pg_stats: echo 'waiting osd.2 seq 64424509453' 2019-12-21T16:05:16.415 INFO:tasks.workunit.client.0.smithi172.stderr:waiting osd.2 seq 64424509453 2019-12-21T16:05:16.415 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 2 2019-12-21T16:05:16.673 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 64424509453 -lt 64424509453 2019-12-21T16:05:16.674 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: flush_pg_stats: for s in $seqs 2019-12-21T16:05:16.674 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: echo 3-197568495618 2019-12-21T16:05:16.674 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: cut -d - -f 1 2019-12-21T16:05:16.674 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: osd=3 2019-12-21T16:05:16.674 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: echo 3-197568495618 2019-12-21T16:05:16.674 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: cut -d - -f 2 2019-12-21T16:05:16.675 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: seq=197568495618 2019-12-21T16:05:16.675 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1989: flush_pg_stats: echo 'waiting osd.3 seq 197568495618' 2019-12-21T16:05:16.675 INFO:tasks.workunit.client.0.smithi172.stderr:waiting osd.3 seq 197568495618 2019-12-21T16:05:16.675 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 3 2019-12-21T16:05:16.900 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 197568495618 -lt 197568495618 2019-12-21T16:05:16.900 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1555: wait_for_clean: get_num_pgs 2019-12-21T16:05:16.900 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: ceph --format json status 2019-12-21T16:05:16.901 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: jq .pgmap.num_pgs 2019-12-21T16:05:17.170 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1555: wait_for_clean: test 5 == 0 2019-12-21T16:05:17.170 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1559: wait_for_clean: true 2019-12-21T16:05:17.171 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1563: wait_for_clean: get_num_active_clean 2019-12-21T16:05:17.171 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: get_num_active_clean: local expression 2019-12-21T16:05:17.171 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2019-12-21T16:05:17.171 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("stale") | not)' 2019-12-21T16:05:17.171 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: ceph --format json pg dump pgs 2019-12-21T16:05:17.171 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2019-12-21T16:05:17.394 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1563: wait_for_clean: cur_active_clean=5 2019-12-21T16:05:17.395 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1564: wait_for_clean: get_num_pgs 2019-12-21T16:05:17.395 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: ceph --format json status 2019-12-21T16:05:17.396 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: jq .pgmap.num_pgs 2019-12-21T16:05:17.664 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1564: wait_for_clean: test 5 = 5 2019-12-21T16:05:17.665 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1564: wait_for_clean: break 2019-12-21T16:05:17.665 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: return 0 2019-12-21T16:05:17.665 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:249: corrupt_and_repair_one: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2019-12-21T16:05:26.767 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:250: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2019-12-21T16:05:26.770 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:267: corrupt_and_repair_erasure_coded: corrupt_and_repair_one td/osd-scrub-repair ecpool 1 2019-12-21T16:05:26.770 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:232: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2019-12-21T16:05:26.770 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:233: corrupt_and_repair_one: local poolname=ecpool 2019-12-21T16:05:26.770 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:234: corrupt_and_repair_one: local osd=1 2019-12-21T16:05:26.771 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:239: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2019-12-21T16:05:26.771 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: objectstore_tool: local dir=td/osd-scrub-repair 2019-12-21T16:05:26.771 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: objectstore_tool: shift 2019-12-21T16:05:26.771 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1235: objectstore_tool: local id=1 2019-12-21T16:05:26.771 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1236: objectstore_tool: shift 2019-12-21T16:05:26.771 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1238: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING remove 2019-12-21T16:05:26.771 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1205: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2019-12-21T16:05:26.772 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nowait: shift 2019-12-21T16:05:26.772 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1207: _objectstore_tool_nowait: local id=1 2019-12-21T16:05:26.772 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1208: _objectstore_tool_nowait: shift 2019-12-21T16:05:26.772 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1210: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2019-12-21T16:05:26.772 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2019-12-21T16:05:26.772 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2019-12-21T16:05:26.772 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2019-12-21T16:05:26.773 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2019-12-21T16:05:26.773 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2019-12-21T16:05:26.877 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2019-12-21T16:05:26.877 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1212: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING remove 2019-12-21T16:05:26.877 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1193: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2019-12-21T16:05:26.878 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1194: _objectstore_tool_nodown: shift 2019-12-21T16:05:26.878 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1195: _objectstore_tool_nodown: local id=1 2019-12-21T16:05:26.878 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: _objectstore_tool_nodown: shift 2019-12-21T16:05:26.878 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2019-12-21T16:05:26.878 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING remove 2019-12-21T16:05:27.437 INFO:tasks.workunit.client.0.smithi172.stdout:remove 1#2:eb822e21:::SOMETHING:head# 2019-12-21T16:05:27.937 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1213: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2019-12-21T16:05:27.938 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:842: activate_osd: local dir=td/osd-scrub-repair 2019-12-21T16:05:27.938 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: shift 2019-12-21T16:05:27.938 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: local id=1 2019-12-21T16:05:27.938 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: shift 2019-12-21T16:05:27.938 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local osd_data=td/osd-scrub-repair/1 2019-12-21T16:05:27.938 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local 'ceph_args=--fsid=973e3fc1-e63e-4bde-9db6-84c68fd5e93e --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2019-12-21T16:05:27.939 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2019-12-21T16:05:27.939 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-journal-size=100' 2019-12-21T16:05:27.939 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2019-12-21T16:05:27.939 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2019-12-21T16:05:27.939 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2019-12-21T16:05:27.939 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --chdir=' 2019-12-21T16:05:27.939 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+= 2019-12-21T16:05:27.940 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2019-12-21T16:05:27.940 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: get_asok_path 2019-12-21T16:05:27.940 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2019-12-21T16:05:27.940 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2019-12-21T16:05:27.940 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2019-12-21T16:05:27.940 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T16:05:27.941 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.119575 2019-12-21T16:05:27.941 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.119575/$cluster-$name.asok' 2019-12-21T16:05:27.941 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.119575/$cluster-$name.asok' 2019-12-21T16:05:27.941 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --debug-osd=20' 2019-12-21T16:05:27.941 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2019-12-21T16:05:27.941 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2019-12-21T16:05:27.941 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2019-12-21T16:05:27.942 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2019-12-21T16:05:27.942 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2019-12-21T16:05:27.942 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' ' 2019-12-21T16:05:27.942 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+= 2019-12-21T16:05:27.942 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: mkdir -p td/osd-scrub-repair/1 2019-12-21T16:05:27.942 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: echo start osd.1 2019-12-21T16:05:27.942 INFO:tasks.workunit.client.0.smithi172.stderr:start osd.1 2019-12-21T16:05:27.943 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph-osd -i 1 --fsid=973e3fc1-e63e-4bde-9db6-84c68fd5e93e --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.119575/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2019-12-21T16:05:27.943 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: cat td/osd-scrub-repair/1/whoami 2019-12-21T16:05:27.943 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: '[' 1 = 1 ']' 2019-12-21T16:05:27.944 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph osd dump --format=json 2019-12-21T16:05:27.944 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: jq '.flags_set[]' 2019-12-21T16:05:27.944 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: grep -q '"noup"' 2019-12-21T16:05:27.958 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:05:27.956+0000 7f68749bcec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T16:05:27.960 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:05:27.958+0000 7f68749bcec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T16:05:27.961 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:05:27.960+0000 7f68749bcec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T16:05:27.963 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:05:27.961+0000 7f68749bcec0 -1 Falling back to public interface 2019-12-21T16:05:28.176 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: wait_for_osd up 1 2019-12-21T16:05:28.176 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:914: wait_for_osd: local state=up 2019-12-21T16:05:28.177 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local id=1 2019-12-21T16:05:28.177 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: status=1 2019-12-21T16:05:28.177 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i=0 )) 2019-12-21T16:05:28.177 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i < 300 )) 2019-12-21T16:05:28.177 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: echo 0 2019-12-21T16:05:28.177 INFO:tasks.workunit.client.0.smithi172.stderr:0 2019-12-21T16:05:28.178 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: ceph osd dump 2019-12-21T16:05:28.178 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: grep 'osd.1 up' 2019-12-21T16:05:28.406 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: sleep 1 2019-12-21T16:05:28.493 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:05:28.492+0000 7f68749bcec0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2019-12-21T16:05:28.501 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:05:28.500+0000 7f68749bcec0 -1 journal do_read_entry(544768): bad header magic 2019-12-21T16:05:28.501 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:05:28.500+0000 7f68749bcec0 -1 journal do_read_entry(544768): bad header magic 2019-12-21T16:05:28.507 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:05:28.506+0000 7f68749bcec0 -1 osd.1 48 log_to_monitors {default=true} 2019-12-21T16:05:29.407 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i++ )) 2019-12-21T16:05:29.408 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i < 300 )) 2019-12-21T16:05:29.408 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: echo 1 2019-12-21T16:05:29.408 INFO:tasks.workunit.client.0.smithi172.stderr:1 2019-12-21T16:05:29.408 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: ceph osd dump 2019-12-21T16:05:29.408 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: grep 'osd.1 up' 2019-12-21T16:05:29.662 INFO:tasks.workunit.client.0.smithi172.stderr:osd.1 up in weight 1 up_from 52 up_thru 52 down_at 49 last_clean_interval [10,48) [v2:127.0.0.1:6810/157742,v1:127.0.0.1:6811/157742] [v2:127.0.0.1:6812/157742,v1:127.0.0.1:6813/157742] exists,up 627d9f35-2c70-4ea0-8821-1ca4dc71496b 2019-12-21T16:05:29.662 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=0 2019-12-21T16:05:29.662 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: break 2019-12-21T16:05:29.662 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: return 0 2019-12-21T16:05:29.663 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1239: objectstore_tool: wait_for_clean 2019-12-21T16:05:29.663 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1548: wait_for_clean: local cmd= 2019-12-21T16:05:29.663 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1549: wait_for_clean: local num_active_clean=-1 2019-12-21T16:05:29.663 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1550: wait_for_clean: local cur_active_clean 2019-12-21T16:05:29.663 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1551: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2019-12-21T16:05:29.663 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1551: wait_for_clean: get_timeout_delays 90 .1 2019-12-21T16:05:29.663 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1494: get_timeout_delays: shopt -q -o xtrace 2019-12-21T16:05:29.664 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1494: get_timeout_delays: echo true 2019-12-21T16:05:29.664 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1494: get_timeout_delays: local trace=true 2019-12-21T16:05:29.664 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1495: get_timeout_delays: true 2019-12-21T16:05:29.664 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1495: get_timeout_delays: shopt -u -o xtrace 2019-12-21T16:05:29.751 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1551: wait_for_clean: local -a delays 2019-12-21T16:05:29.752 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1552: wait_for_clean: local -i loop=0 2019-12-21T16:05:29.752 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1554: wait_for_clean: flush_pg_stats 2019-12-21T16:05:29.752 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1977: flush_pg_stats: local timeout=300 2019-12-21T16:05:29.752 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1979: flush_pg_stats: ceph osd ls 2019-12-21T16:05:29.978 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1979: flush_pg_stats: ids='0 2019-12-21T16:05:29.978 INFO:tasks.workunit.client.0.smithi172.stderr:1 2019-12-21T16:05:29.979 INFO:tasks.workunit.client.0.smithi172.stderr:2 2019-12-21T16:05:29.979 INFO:tasks.workunit.client.0.smithi172.stderr:3' 2019-12-21T16:05:29.979 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1980: flush_pg_stats: seqs= 2019-12-21T16:05:29.979 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: flush_pg_stats: for osd in $ids 2019-12-21T16:05:29.979 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2019-12-21T16:05:30.087 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: seq=21474836499 2019-12-21T16:05:30.088 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: flush_pg_stats: seqs=' 0-21474836499' 2019-12-21T16:05:30.088 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: flush_pg_stats: for osd in $ids 2019-12-21T16:05:30.088 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2019-12-21T16:05:30.196 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: seq=223338299394 2019-12-21T16:05:30.196 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: flush_pg_stats: seqs=' 0-21474836499 1-223338299394' 2019-12-21T16:05:30.197 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: flush_pg_stats: for osd in $ids 2019-12-21T16:05:30.197 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2019-12-21T16:05:30.304 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: seq=64424509458 2019-12-21T16:05:30.305 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: flush_pg_stats: seqs=' 0-21474836499 1-223338299394 2-64424509458' 2019-12-21T16:05:30.305 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: flush_pg_stats: for osd in $ids 2019-12-21T16:05:30.305 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2019-12-21T16:05:30.413 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: seq=197568495622 2019-12-21T16:05:30.413 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: flush_pg_stats: seqs=' 0-21474836499 1-223338299394 2-64424509458 3-197568495622' 2019-12-21T16:05:30.414 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: flush_pg_stats: for s in $seqs 2019-12-21T16:05:30.414 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: echo 0-21474836499 2019-12-21T16:05:30.414 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: cut -d - -f 1 2019-12-21T16:05:30.415 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: osd=0 2019-12-21T16:05:30.415 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: echo 0-21474836499 2019-12-21T16:05:30.415 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: cut -d - -f 2 2019-12-21T16:05:30.416 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: seq=21474836499 2019-12-21T16:05:30.416 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1989: flush_pg_stats: echo 'waiting osd.0 seq 21474836499' 2019-12-21T16:05:30.416 INFO:tasks.workunit.client.0.smithi172.stderr:waiting osd.0 seq 21474836499 2019-12-21T16:05:30.417 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 0 2019-12-21T16:05:30.643 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 21474836498 -lt 21474836499 2019-12-21T16:05:30.643 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1991: flush_pg_stats: sleep 1 2019-12-21T16:05:31.645 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1992: flush_pg_stats: '[' 300 -eq 0 ']' 2019-12-21T16:05:31.645 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 0 2019-12-21T16:05:31.880 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 21474836499 -lt 21474836499 2019-12-21T16:05:31.881 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: flush_pg_stats: for s in $seqs 2019-12-21T16:05:31.881 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: echo 1-223338299394 2019-12-21T16:05:31.881 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: cut -d - -f 1 2019-12-21T16:05:31.882 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: osd=1 2019-12-21T16:05:31.882 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: echo 1-223338299394 2019-12-21T16:05:31.883 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: cut -d - -f 2 2019-12-21T16:05:31.883 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: seq=223338299394 2019-12-21T16:05:31.883 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1989: flush_pg_stats: echo 'waiting osd.1 seq 223338299394' 2019-12-21T16:05:31.883 INFO:tasks.workunit.client.0.smithi172.stderr:waiting osd.1 seq 223338299394 2019-12-21T16:05:31.884 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 1 2019-12-21T16:05:32.109 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 223338299394 -lt 223338299394 2019-12-21T16:05:32.110 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: flush_pg_stats: for s in $seqs 2019-12-21T16:05:32.110 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: echo 2-64424509458 2019-12-21T16:05:32.110 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: cut -d - -f 1 2019-12-21T16:05:32.111 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: osd=2 2019-12-21T16:05:32.111 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: echo 2-64424509458 2019-12-21T16:05:32.112 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: cut -d - -f 2 2019-12-21T16:05:32.114 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: seq=64424509458 2019-12-21T16:05:32.115 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1989: flush_pg_stats: echo 'waiting osd.2 seq 64424509458' 2019-12-21T16:05:32.115 INFO:tasks.workunit.client.0.smithi172.stderr:waiting osd.2 seq 64424509458 2019-12-21T16:05:32.115 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 2 2019-12-21T16:05:32.339 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 64424509458 -lt 64424509458 2019-12-21T16:05:32.339 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: flush_pg_stats: for s in $seqs 2019-12-21T16:05:32.339 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: echo 3-197568495622 2019-12-21T16:05:32.340 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: cut -d - -f 1 2019-12-21T16:05:32.340 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: osd=3 2019-12-21T16:05:32.341 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: echo 3-197568495622 2019-12-21T16:05:32.341 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: cut -d - -f 2 2019-12-21T16:05:32.342 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: seq=197568495622 2019-12-21T16:05:32.342 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1989: flush_pg_stats: echo 'waiting osd.3 seq 197568495622' 2019-12-21T16:05:32.342 INFO:tasks.workunit.client.0.smithi172.stderr:waiting osd.3 seq 197568495622 2019-12-21T16:05:32.342 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 3 2019-12-21T16:05:32.568 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 197568495622 -lt 197568495622 2019-12-21T16:05:32.569 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1555: wait_for_clean: get_num_pgs 2019-12-21T16:05:32.569 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: ceph --format json status 2019-12-21T16:05:32.569 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: jq .pgmap.num_pgs 2019-12-21T16:05:32.838 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1555: wait_for_clean: test 5 == 0 2019-12-21T16:05:32.838 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1559: wait_for_clean: true 2019-12-21T16:05:32.838 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1563: wait_for_clean: get_num_active_clean 2019-12-21T16:05:32.839 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: get_num_active_clean: local expression 2019-12-21T16:05:32.839 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2019-12-21T16:05:32.839 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("stale") | not)' 2019-12-21T16:05:32.839 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: ceph --format json pg dump pgs 2019-12-21T16:05:32.839 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2019-12-21T16:05:33.061 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1563: wait_for_clean: cur_active_clean=5 2019-12-21T16:05:33.062 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1564: wait_for_clean: get_num_pgs 2019-12-21T16:05:33.062 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: ceph --format json status 2019-12-21T16:05:33.062 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: jq .pgmap.num_pgs 2019-12-21T16:05:33.330 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1564: wait_for_clean: test 5 = 5 2019-12-21T16:05:33.331 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1564: wait_for_clean: break 2019-12-21T16:05:33.331 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: return 0 2019-12-21T16:05:33.331 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:243: corrupt_and_repair_one: get_pg ecpool SOMETHING 2019-12-21T16:05:33.331 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1029: get_pg: local poolname=ecpool 2019-12-21T16:05:33.331 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1030: get_pg: local objectname=SOMETHING 2019-12-21T16:05:33.332 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1032: get_pg: ceph --format json osd map ecpool SOMETHING 2019-12-21T16:05:33.332 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1032: get_pg: jq -r .pgid 2019-12-21T16:05:33.572 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:243: corrupt_and_repair_one: local pg=2.0 2019-12-21T16:05:33.572 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:244: corrupt_and_repair_one: repair 2.0 2019-12-21T16:05:33.573 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: repair: local pgid=2.0 2019-12-21T16:05:33.573 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1665: repair: get_last_scrub_stamp 2.0 2019-12-21T16:05:33.574 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=2.0 2019-12-21T16:05:33.574 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T16:05:33.574 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T16:05:33.574 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2019-12-21T16:05:33.815 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1665: repair: local last_scrub=2019-12-21T16:05:06.199497+0000 2019-12-21T16:05:33.815 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1666: repair: ceph pg repair 2.0 2019-12-21T16:05:34.028 INFO:tasks.workunit.client.0.smithi172.stderr:instructing pg 2.0s0 on osd.3 to repair 2019-12-21T16:05:34.037 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: repair: wait_for_scrub 2.0 2019-12-21T16:05:06.199497+0000 2019-12-21T16:05:34.037 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1789: wait_for_scrub: local pgid=2.0 2019-12-21T16:05:34.037 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1790: wait_for_scrub: local last_scrub=2019-12-21T16:05:06.199497+0000 2019-12-21T16:05:34.038 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1791: wait_for_scrub: local sname=last_scrub_stamp 2019-12-21T16:05:34.038 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i=0 )) 2019-12-21T16:05:34.038 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T16:05:34.038 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2019-12-21T16:05:34.038 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=2.0 2019-12-21T16:05:34.038 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T16:05:34.038 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T16:05:34.039 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2019-12-21T16:05:34.260 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T16:05:06.199497+0000 '>' 2019-12-21T16:05:06.199497+0000 2019-12-21T16:05:34.260 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T16:05:35.261 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T16:05:35.262 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T16:05:35.262 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2019-12-21T16:05:35.262 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=2.0 2019-12-21T16:05:35.263 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T16:05:35.263 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T16:05:35.263 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2019-12-21T16:05:35.493 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T16:05:06.199497+0000 '>' 2019-12-21T16:05:06.199497+0000 2019-12-21T16:05:35.493 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T16:05:36.494 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T16:05:36.494 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T16:05:36.495 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2019-12-21T16:05:36.495 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=2.0 2019-12-21T16:05:36.495 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T16:05:36.495 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T16:05:36.496 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2019-12-21T16:05:36.728 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T16:05:06.199497+0000 '>' 2019-12-21T16:05:06.199497+0000 2019-12-21T16:05:36.728 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T16:05:37.729 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T16:05:37.729 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T16:05:37.730 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2019-12-21T16:05:37.730 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=2.0 2019-12-21T16:05:37.730 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T16:05:37.730 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T16:05:37.731 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2019-12-21T16:05:37.961 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T16:05:35.219372+0000 '>' 2019-12-21T16:05:06.199497+0000 2019-12-21T16:05:37.962 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1795: wait_for_scrub: return 0 2019-12-21T16:05:37.962 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:248: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2019-12-21T16:05:37.962 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: objectstore_tool: local dir=td/osd-scrub-repair 2019-12-21T16:05:37.962 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: objectstore_tool: shift 2019-12-21T16:05:37.962 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1235: objectstore_tool: local id=3 2019-12-21T16:05:37.962 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1236: objectstore_tool: shift 2019-12-21T16:05:37.962 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1238: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING list-attrs 2019-12-21T16:05:37.963 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1205: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2019-12-21T16:05:37.963 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nowait: shift 2019-12-21T16:05:37.963 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1207: _objectstore_tool_nowait: local id=3 2019-12-21T16:05:37.963 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1208: _objectstore_tool_nowait: shift 2019-12-21T16:05:37.963 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1210: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2019-12-21T16:05:37.963 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2019-12-21T16:05:37.963 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2019-12-21T16:05:37.964 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2019-12-21T16:05:37.964 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2019-12-21T16:05:37.964 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2019-12-21T16:05:38.067 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2019-12-21T16:05:38.067 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1212: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING list-attrs 2019-12-21T16:05:38.067 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1193: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2019-12-21T16:05:38.068 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1194: _objectstore_tool_nodown: shift 2019-12-21T16:05:38.068 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1195: _objectstore_tool_nodown: local id=3 2019-12-21T16:05:38.068 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: _objectstore_tool_nodown: shift 2019-12-21T16:05:38.068 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2019-12-21T16:05:38.068 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING list-attrs 2019-12-21T16:05:38.690 INFO:tasks.workunit.client.0.smithi172.stderr:Error getting attr on : 2.0s0_head,0#-4:00000000:::scrub_2.0s0:head#, (61) No data available 2019-12-21T16:05:38.690 INFO:tasks.workunit.client.0.smithi172.stdout:_ 2019-12-21T16:05:38.690 INFO:tasks.workunit.client.0.smithi172.stdout:hinfo_key 2019-12-21T16:05:38.690 INFO:tasks.workunit.client.0.smithi172.stdout:snapset 2019-12-21T16:05:38.769 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1213: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2019-12-21T16:05:38.769 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:842: activate_osd: local dir=td/osd-scrub-repair 2019-12-21T16:05:38.769 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: shift 2019-12-21T16:05:38.769 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: local id=3 2019-12-21T16:05:38.769 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: shift 2019-12-21T16:05:38.770 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local osd_data=td/osd-scrub-repair/3 2019-12-21T16:05:38.770 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local 'ceph_args=--fsid=973e3fc1-e63e-4bde-9db6-84c68fd5e93e --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2019-12-21T16:05:38.770 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2019-12-21T16:05:38.770 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-journal-size=100' 2019-12-21T16:05:38.770 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2019-12-21T16:05:38.770 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2019-12-21T16:05:38.770 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2019-12-21T16:05:38.771 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --chdir=' 2019-12-21T16:05:38.771 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+= 2019-12-21T16:05:38.771 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2019-12-21T16:05:38.771 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: get_asok_path 2019-12-21T16:05:38.771 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2019-12-21T16:05:38.771 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2019-12-21T16:05:38.771 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2019-12-21T16:05:38.772 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T16:05:38.772 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.119575 2019-12-21T16:05:38.772 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.119575/$cluster-$name.asok' 2019-12-21T16:05:38.772 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.119575/$cluster-$name.asok' 2019-12-21T16:05:38.772 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --debug-osd=20' 2019-12-21T16:05:38.772 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2019-12-21T16:05:38.773 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2019-12-21T16:05:38.773 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2019-12-21T16:05:38.773 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2019-12-21T16:05:38.773 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2019-12-21T16:05:38.773 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' ' 2019-12-21T16:05:38.773 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+= 2019-12-21T16:05:38.773 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: mkdir -p td/osd-scrub-repair/3 2019-12-21T16:05:38.773 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: echo start osd.3 2019-12-21T16:05:38.774 INFO:tasks.workunit.client.0.smithi172.stderr:start osd.3 2019-12-21T16:05:38.774 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph-osd -i 3 --fsid=973e3fc1-e63e-4bde-9db6-84c68fd5e93e --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.119575/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2019-12-21T16:05:38.774 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: cat td/osd-scrub-repair/3/whoami 2019-12-21T16:05:38.775 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: '[' 3 = 3 ']' 2019-12-21T16:05:38.775 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph osd dump --format=json 2019-12-21T16:05:38.775 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: jq '.flags_set[]' 2019-12-21T16:05:38.775 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: grep -q '"noup"' 2019-12-21T16:05:38.787 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:05:38.786+0000 7fbe164f1ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T16:05:38.941 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:05:38.939+0000 7fbe164f1ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T16:05:38.944 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:05:38.943+0000 7fbe164f1ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T16:05:38.945 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:05:38.944+0000 7fbe164f1ec0 -1 Falling back to public interface 2019-12-21T16:05:39.094 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: wait_for_osd up 3 2019-12-21T16:05:39.094 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:914: wait_for_osd: local state=up 2019-12-21T16:05:39.094 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local id=3 2019-12-21T16:05:39.094 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: status=1 2019-12-21T16:05:39.095 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i=0 )) 2019-12-21T16:05:39.095 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i < 300 )) 2019-12-21T16:05:39.095 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: echo 0 2019-12-21T16:05:39.095 INFO:tasks.workunit.client.0.smithi172.stderr:0 2019-12-21T16:05:39.095 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: ceph osd dump 2019-12-21T16:05:39.095 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: grep 'osd.3 up' 2019-12-21T16:05:39.373 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: sleep 1 2019-12-21T16:05:39.611 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:05:39.609+0000 7fbe164f1ec0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2019-12-21T16:05:39.619 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:05:39.617+0000 7fbe164f1ec0 -1 journal do_read_entry(696320): bad header magic 2019-12-21T16:05:39.619 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:05:39.617+0000 7fbe164f1ec0 -1 journal do_read_entry(696320): bad header magic 2019-12-21T16:05:39.625 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:05:39.623+0000 7fbe164f1ec0 -1 osd.3 54 log_to_monitors {default=true} 2019-12-21T16:05:40.375 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i++ )) 2019-12-21T16:05:40.375 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i < 300 )) 2019-12-21T16:05:40.375 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: echo 1 2019-12-21T16:05:40.375 INFO:tasks.workunit.client.0.smithi172.stderr:1 2019-12-21T16:05:40.376 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: ceph osd dump 2019-12-21T16:05:40.376 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: grep 'osd.3 up' 2019-12-21T16:05:40.604 INFO:tasks.workunit.client.0.smithi172.stderr:osd.3 up in weight 1 up_from 58 up_thru 58 down_at 55 last_clean_interval [46,54) [v2:127.0.0.1:6826/158645,v1:127.0.0.1:6827/158645] [v2:127.0.0.1:6828/158645,v1:127.0.0.1:6829/158645] exists,up d2b8f48d-5699-449e-8fc7-707402a64a19 2019-12-21T16:05:40.605 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=0 2019-12-21T16:05:40.605 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: break 2019-12-21T16:05:40.605 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: return 0 2019-12-21T16:05:40.605 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1239: objectstore_tool: wait_for_clean 2019-12-21T16:05:40.605 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1548: wait_for_clean: local cmd= 2019-12-21T16:05:40.605 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1549: wait_for_clean: local num_active_clean=-1 2019-12-21T16:05:40.606 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1550: wait_for_clean: local cur_active_clean 2019-12-21T16:05:40.606 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1551: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2019-12-21T16:05:40.606 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1551: wait_for_clean: get_timeout_delays 90 .1 2019-12-21T16:05:40.606 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1494: get_timeout_delays: shopt -q -o xtrace 2019-12-21T16:05:40.606 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1494: get_timeout_delays: echo true 2019-12-21T16:05:40.606 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1494: get_timeout_delays: local trace=true 2019-12-21T16:05:40.606 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1495: get_timeout_delays: true 2019-12-21T16:05:40.607 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1495: get_timeout_delays: shopt -u -o xtrace 2019-12-21T16:05:40.697 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1551: wait_for_clean: local -a delays 2019-12-21T16:05:40.697 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1552: wait_for_clean: local -i loop=0 2019-12-21T16:05:40.698 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1554: wait_for_clean: flush_pg_stats 2019-12-21T16:05:40.698 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1977: flush_pg_stats: local timeout=300 2019-12-21T16:05:40.698 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1979: flush_pg_stats: ceph osd ls 2019-12-21T16:05:40.924 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1979: flush_pg_stats: ids='0 2019-12-21T16:05:40.924 INFO:tasks.workunit.client.0.smithi172.stderr:1 2019-12-21T16:05:40.925 INFO:tasks.workunit.client.0.smithi172.stderr:2 2019-12-21T16:05:40.925 INFO:tasks.workunit.client.0.smithi172.stderr:3' 2019-12-21T16:05:40.925 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1980: flush_pg_stats: seqs= 2019-12-21T16:05:40.925 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: flush_pg_stats: for osd in $ids 2019-12-21T16:05:40.925 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2019-12-21T16:05:41.106 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: seq=21474836502 2019-12-21T16:05:41.106 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: flush_pg_stats: seqs=' 0-21474836502' 2019-12-21T16:05:41.106 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: flush_pg_stats: for osd in $ids 2019-12-21T16:05:41.107 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2019-12-21T16:05:41.215 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: seq=223338299397 2019-12-21T16:05:41.216 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: flush_pg_stats: seqs=' 0-21474836502 1-223338299397' 2019-12-21T16:05:41.216 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: flush_pg_stats: for osd in $ids 2019-12-21T16:05:41.216 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2019-12-21T16:05:41.324 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: seq=64424509461 2019-12-21T16:05:41.324 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: flush_pg_stats: seqs=' 0-21474836502 1-223338299397 2-64424509461' 2019-12-21T16:05:41.324 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: flush_pg_stats: for osd in $ids 2019-12-21T16:05:41.325 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2019-12-21T16:05:41.433 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: seq=249108103170 2019-12-21T16:05:41.434 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: flush_pg_stats: seqs=' 0-21474836502 1-223338299397 2-64424509461 3-249108103170' 2019-12-21T16:05:41.434 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: flush_pg_stats: for s in $seqs 2019-12-21T16:05:41.435 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: echo 0-21474836502 2019-12-21T16:05:41.435 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: cut -d - -f 1 2019-12-21T16:05:41.435 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: osd=0 2019-12-21T16:05:41.436 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: echo 0-21474836502 2019-12-21T16:05:41.436 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: cut -d - -f 2 2019-12-21T16:05:41.437 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: seq=21474836502 2019-12-21T16:05:41.437 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1989: flush_pg_stats: echo 'waiting osd.0 seq 21474836502' 2019-12-21T16:05:41.437 INFO:tasks.workunit.client.0.smithi172.stderr:waiting osd.0 seq 21474836502 2019-12-21T16:05:41.437 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 0 2019-12-21T16:05:41.662 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 21474836501 -lt 21474836502 2019-12-21T16:05:41.663 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1991: flush_pg_stats: sleep 1 2019-12-21T16:05:42.664 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1992: flush_pg_stats: '[' 300 -eq 0 ']' 2019-12-21T16:05:42.664 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 0 2019-12-21T16:05:42.899 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 21474836501 -lt 21474836502 2019-12-21T16:05:42.899 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1991: flush_pg_stats: sleep 1 2019-12-21T16:05:43.900 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1992: flush_pg_stats: '[' 299 -eq 0 ']' 2019-12-21T16:05:43.900 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 0 2019-12-21T16:05:44.139 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 21474836502 -lt 21474836502 2019-12-21T16:05:44.139 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: flush_pg_stats: for s in $seqs 2019-12-21T16:05:44.139 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: echo 1-223338299397 2019-12-21T16:05:44.139 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: cut -d - -f 1 2019-12-21T16:05:44.139 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: osd=1 2019-12-21T16:05:44.140 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: echo 1-223338299397 2019-12-21T16:05:44.140 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: cut -d - -f 2 2019-12-21T16:05:44.141 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: seq=223338299397 2019-12-21T16:05:44.142 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1989: flush_pg_stats: echo 'waiting osd.1 seq 223338299397' 2019-12-21T16:05:44.142 INFO:tasks.workunit.client.0.smithi172.stderr:waiting osd.1 seq 223338299397 2019-12-21T16:05:44.142 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 1 2019-12-21T16:05:44.416 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 223338299397 -lt 223338299397 2019-12-21T16:05:44.417 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: flush_pg_stats: for s in $seqs 2019-12-21T16:05:44.417 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: echo 2-64424509461 2019-12-21T16:05:44.417 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: cut -d - -f 1 2019-12-21T16:05:44.418 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: osd=2 2019-12-21T16:05:44.419 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: echo 2-64424509461 2019-12-21T16:05:44.419 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: cut -d - -f 2 2019-12-21T16:05:44.420 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: seq=64424509461 2019-12-21T16:05:44.420 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1989: flush_pg_stats: echo 'waiting osd.2 seq 64424509461' 2019-12-21T16:05:44.420 INFO:tasks.workunit.client.0.smithi172.stderr:waiting osd.2 seq 64424509461 2019-12-21T16:05:44.420 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 2 2019-12-21T16:05:44.647 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 64424509461 -lt 64424509461 2019-12-21T16:05:44.647 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: flush_pg_stats: for s in $seqs 2019-12-21T16:05:44.648 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: echo 3-249108103170 2019-12-21T16:05:44.648 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: cut -d - -f 1 2019-12-21T16:05:44.649 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: osd=3 2019-12-21T16:05:44.649 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: echo 3-249108103170 2019-12-21T16:05:44.649 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: cut -d - -f 2 2019-12-21T16:05:44.650 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: seq=249108103170 2019-12-21T16:05:44.650 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1989: flush_pg_stats: echo 'waiting osd.3 seq 249108103170' 2019-12-21T16:05:44.650 INFO:tasks.workunit.client.0.smithi172.stderr:waiting osd.3 seq 249108103170 2019-12-21T16:05:44.651 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 3 2019-12-21T16:05:44.910 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 249108103170 -lt 249108103170 2019-12-21T16:05:44.911 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1555: wait_for_clean: get_num_pgs 2019-12-21T16:05:44.911 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: ceph --format json status 2019-12-21T16:05:44.911 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: jq .pgmap.num_pgs 2019-12-21T16:05:45.180 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1555: wait_for_clean: test 5 == 0 2019-12-21T16:05:45.180 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1559: wait_for_clean: true 2019-12-21T16:05:45.180 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1563: wait_for_clean: get_num_active_clean 2019-12-21T16:05:45.180 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: get_num_active_clean: local expression 2019-12-21T16:05:45.180 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2019-12-21T16:05:45.181 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("stale") | not)' 2019-12-21T16:05:45.181 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: ceph --format json pg dump pgs 2019-12-21T16:05:45.181 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2019-12-21T16:05:45.426 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1563: wait_for_clean: cur_active_clean=5 2019-12-21T16:05:45.426 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1564: wait_for_clean: get_num_pgs 2019-12-21T16:05:45.427 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: ceph --format json status 2019-12-21T16:05:45.427 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: jq .pgmap.num_pgs 2019-12-21T16:05:45.695 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1564: wait_for_clean: test 5 = 5 2019-12-21T16:05:45.695 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1564: wait_for_clean: break 2019-12-21T16:05:45.695 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: return 0 2019-12-21T16:05:45.696 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:249: corrupt_and_repair_one: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2019-12-21T16:05:54.574 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:250: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2019-12-21T16:05:54.577 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:268: corrupt_and_repair_erasure_coded: corrupt_and_repair_two td/osd-scrub-repair ecpool 1 2 2019-12-21T16:05:54.577 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:191: corrupt_and_repair_two: local dir=td/osd-scrub-repair 2019-12-21T16:05:54.577 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:192: corrupt_and_repair_two: local poolname=ecpool 2019-12-21T16:05:54.578 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:193: corrupt_and_repair_two: local first=1 2019-12-21T16:05:54.578 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:194: corrupt_and_repair_two: local second=2 2019-12-21T16:05:54.578 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:199: corrupt_and_repair_two: pids= 2019-12-21T16:05:54.578 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:200: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2019-12-21T16:05:54.579 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1907: run_in_background: local pid_variable=pids 2019-12-21T16:05:54.579 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1908: run_in_background: shift 2019-12-21T16:05:54.579 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1912: run_in_background: eval 'pids+=" 159291"' 2019-12-21T16:05:54.579 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1912: run_in_background: pids+=' 159291' 2019-12-21T16:05:54.579 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:201: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 2 SOMETHING remove 2019-12-21T16:05:54.579 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1907: run_in_background: local pid_variable=pids 2019-12-21T16:05:54.579 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1908: run_in_background: shift 2019-12-21T16:05:54.580 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1911: run_in_background: objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2019-12-21T16:05:54.580 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1912: run_in_background: eval 'pids+=" 159293"' 2019-12-21T16:05:54.580 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1912: run_in_background: pids+=' 159293' 2019-12-21T16:05:54.580 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:202: corrupt_and_repair_two: wait_background pids 2019-12-21T16:05:54.580 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1943: wait_background: pids=' 159291 159293' 2019-12-21T16:05:54.580 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1945: wait_background: return_code=0 2019-12-21T16:05:54.580 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1946: wait_background: for pid in $pids 2019-12-21T16:05:54.581 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1947: wait_background: wait 159291 2019-12-21T16:05:54.581 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1911: run_in_background: sed 's/^/119575: /' 2019-12-21T16:05:54.581 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1911: run_in_background: objectstore_tool td/osd-scrub-repair 2 SOMETHING remove 2019-12-21T16:05:54.581 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1911: run_in_background: sed 's/^/119575: /' 2019-12-21T16:05:56.027 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: objectstore_tool: local dir=td/osd-scrub-repair 2019-12-21T16:05:56.028 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: objectstore_tool: shift 2019-12-21T16:05:56.028 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1235: objectstore_tool: local id=1 2019-12-21T16:05:56.028 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1236: objectstore_tool: shift 2019-12-21T16:05:56.028 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1238: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING remove 2019-12-21T16:05:56.028 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1205: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2019-12-21T16:05:56.028 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nowait: shift 2019-12-21T16:05:56.028 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1207: _objectstore_tool_nowait: local id=1 2019-12-21T16:05:56.029 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1208: _objectstore_tool_nowait: shift 2019-12-21T16:05:56.029 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1210: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2019-12-21T16:05:56.029 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2019-12-21T16:05:56.029 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2019-12-21T16:05:56.029 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2019-12-21T16:05:56.029 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2019-12-21T16:05:56.029 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2019-12-21T16:05:56.030 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2019-12-21T16:05:56.030 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1212: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING remove 2019-12-21T16:05:56.030 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1193: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2019-12-21T16:05:56.030 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1194: _objectstore_tool_nodown: shift 2019-12-21T16:05:56.030 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1195: _objectstore_tool_nodown: local id=1 2019-12-21T16:05:56.030 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: _objectstore_tool_nodown: shift 2019-12-21T16:05:56.031 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2019-12-21T16:05:56.031 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING remove 2019-12-21T16:05:56.031 INFO:tasks.workunit.client.0.smithi172.stderr:119575: remove 1#2:eb822e21:::SOMETHING:head# 2019-12-21T16:05:56.031 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1213: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2019-12-21T16:05:56.031 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:842: activate_osd: local dir=td/osd-scrub-repair 2019-12-21T16:05:56.031 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: shift 2019-12-21T16:05:56.031 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: local id=1 2019-12-21T16:05:56.032 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: shift 2019-12-21T16:05:56.032 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local osd_data=td/osd-scrub-repair/1 2019-12-21T16:05:56.032 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local 'ceph_args=--fsid=973e3fc1-e63e-4bde-9db6-84c68fd5e93e --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2019-12-21T16:05:56.032 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2019-12-21T16:05:56.032 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: 119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: objectstore_tool: local dir=td/osd-scrub-repair 2019-12-21T16:05:56.035 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: objectstore_tool: shift 2019-12-21T16:05:56.035 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1235: objectstore_tool: local id=2 2019-12-21T16:05:56.035 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1236: objectstore_tool: shift 2019-12-21T16:05:56.035 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1238: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 2 SOMETHING remove 2019-12-21T16:05:56.035 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1205: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2019-12-21T16:05:56.036 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nowait: shift 2019-12-21T16:05:56.036 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1207: _objectstore_tool_nowait: local id=2 2019-12-21T16:05:56.037 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1208: _objectstore_tool_nowait: shift 2019-12-21T16:05:56.037 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1210: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.2 2019-12-21T16:05:56.037 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2019-12-21T16:05:56.038 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2019-12-21T16:05:56.038 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2019-12-21T16:05:56.038 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2019-12-21T16:05:56.038 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2019-12-21T16:05:56.038 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2019-12-21T16:05:56.038 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1212: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 2 SOMETHING remove 2019-12-21T16:05:56.038 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1193: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2019-12-21T16:05:56.039 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1194: _objectstore_tool_nodown: shift 2019-12-21T16:05:56.039 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1195: _objectstore_tool_nodown: local id=2 2019-12-21T16:05:56.039 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: _objectstore_tool_nodown: shift 2019-12-21T16:05:56.039 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/2 2019-12-21T16:05:56.039 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 SOMETHING remove 2019-12-21T16:05:56.039 INFO:tasks.workunit.client.0.smithi172.stderr:119575: remove 2#2:eb822e21:::SOMETHING:head# 2019-12-21T16:05:56.040 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1213: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 2 2019-12-21T16:05:56.040 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:842: activate_osd: local dir=td/osd-scrub-repair 2019-12-21T16:05:56.040 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: shift 2019-12-21T16:05:56.040 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: local id=2 2019-12-21T16:05:56.040 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: shift 2019-12-21T16:05:56.040 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local osd_data=td/osd-scrub-repair/2 2019-12-21T16:05:56.040 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local 'ceph_args=--fsid=973e3fc1-e63e-4bde-9db6-84c68fd5e93e --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2019-12-21T16:05:56.041 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2019-12-21T16:05:56.041 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-journal-size=100' 2019-12-21T16:05:56.041 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2019-12-21T16:05:56.041 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2019-12-21T16:05:56.042 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2019-12-21T16:05:56.042 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --chdir=' 2019-12-21T16:05:56.042 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+= 2019-12-21T16:05:56.042 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2019-12-21T16:05:56.042 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: get_asok_path 2019-12-21T16:05:56.042 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2019-12-21T16:05:56.042 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2019-12-21T16:05:56.043 INFO:tasks.workunit.client.0.smithi172.stderr:119575: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2019-12-21T16:05:56.043 INFO:tasks.workunit.client.0.smithi172.stderr:119575: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T16:05:56.043 INFO:tasks.workunit.client.0.smithi172.stderr:119575: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.119575 2019-12-21T16:05:56.043 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.119575/$cluster-$name.asok' 2019-12-21T16:05:56.043 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.119575/$cluster-$name.asok' 2019-12-21T16:05:56.043 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --debug-osd=20' 2019-12-21T16:05:56.044 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2019-12-21T16:05:56.044 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2019-12-21T16:05:56.044 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2019-12-21T16:05:56.044 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2019-12-21T16:05:56.060 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2019-12-21T16:05:56.060 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' ' 2019-12-21T16:05:56.061 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+= 2019-12-21T16:05:56.061 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: mkdir -p td/osd-scrub-repair/1 2019-12-21T16:05:56.061 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: echo start osd.1 2019-12-21T16:05:56.061 INFO:tasks.workunit.client.0.smithi172.stderr:119575: start osd.1 2019-12-21T16:05:56.061 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph-osd -i 1 --fsid=973e3fc1-e63e-4bde-9db6-84c68fd5e93e --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.119575/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2019-12-21T16:05:56.076 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: cat td/osd-scrub-repair/1/whoami 2019-12-21T16:05:56.076 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standaactivate_osd: ceph_args+=' --osd-journal-size=100' 2019-12-21T16:05:56.077 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2019-12-21T16:05:56.077 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2019-12-21T16:05:56.077 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2019-12-21T16:05:56.077 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --chdir=' 2019-12-21T16:05:56.077 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+= 2019-12-21T16:05:56.077 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2019-12-21T16:05:56.078 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: get_asok_path 2019-12-21T16:05:56.078 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2019-12-21T16:05:56.078 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2019-12-21T16:05:56.078 INFO:tasks.workunit.client.0.smithi172.stderr:119575: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2019-12-21T16:05:56.078 INFO:tasks.workunit.client.0.smithi172.stderr:119575: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T16:05:56.078 INFO:tasks.workunit.client.0.smithi172.stderr:119575: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.119575 2019-12-21T16:05:56.078 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.119575/$cluster-$name.asok' 2019-12-21T16:05:56.079 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.119575/$cluster-$name.asok' 2019-12-21T16:05:56.079 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --debug-osd=20' 2019-12-21T16:05:56.079 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2019-12-21T16:05:56.079 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2019-12-21T16:05:56.079 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2019-12-21T16:05:56.079 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2019-12-21T16:05:56.079 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2019-12-21T16:05:56.080 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' ' 2019-12-21T16:05:56.080 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+= 2019-12-21T16:05:56.080 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: mkdir -p td/osd-scrub-repair/2 2019-12-21T16:05:56.080 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: echo start osd.2 2019-12-21T16:05:56.080 INFO:tasks.workunit.client.0.smithi172.stderr:119575: start osd.2 2019-12-21T16:05:56.080 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph-osd -i 2 --fsid=973e3fc1-e63e-4bde-9db6-84c68fd5e93e --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.119575/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2019-12-21T16:05:56.080 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: cat td/osd-scrub-repair/2/whoami 2019-12-21T16:05:56.659 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: '[' 2 = 2 ']' 2019-12-21T16:05:56.659 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: jq '.flags_set[]' 2019-12-21T16:05:56.659 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: grep -q '"noup"' 2019-12-21T16:05:56.659 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph osd dump --format=json 2019-12-21T16:05:56.659 INFO:tasks.workunit.client.0.smithi172.stderr:119575: 2019-12-21T16:05:56.050+0000 7f1c9c0b8ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T16:05:56.659 INFO:tasks.workunit.client.0.smithi172.stderr:119575: 2019-12-21T16:05:56.052+0000 7f1c9c0b8ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T16:05:56.660 INFO:tasks.workunit.client.0.smithi172.stderr:119575: 2019-12-21T16:05:56.054+0000 7f1c9c0b8ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T16:05:56.660 INFO:tasks.workunit.client.0.smithi172.stderr:119575: 2019-12-21T16:05:56.055+0000 7f1c9c0b8ec0 -1 Falling back to public interface 2019-12-21T16:05:56.660 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: wait_for_osd up 2 2019-12-21T16:05:56.660 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:914: wait_for_osd: local state=up 2019-12-21T16:05:56.660 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local id=2 2019-12-21T16:05:56.660 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: status=1 2019-12-21T16:05:56.661 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i=0 )) 2019-12-21T16:05:56.661 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i < 300 )) 2019-12-21T16:05:56.661 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: echo 0 2019-12-21T16:05:56.661 INFO:tasks.workunit.client.0.smithi172.stderr:119575: 0 2019-12-21T16:05:56.661 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: ceph osd dump 2019-12-21T16:05:56.661 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: grep 'osd.2 up' 2019-12-21T16:05:56.661 INFO:tasks.workunit.client.0.smithi172.stderr:119575: 2019-12-21T16:05:56.439+0000 7f1c9c0b8ec0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2019-12-21T16:05:56.662 INFO:tasks.workunit.client.0.smithi172.stderr:119575: 2019-12-21T16:05:56.448+0000 7f1c9c0b8ec0 -1 journal do_read_entry(753664): bad header magic 2019-12-21T16:05:56.662 INFO:tasks.workunit.client.0.smithi172.stderr:119575: 2019-12-21T16:05:56.448+0000 7f1c9c0b8ec0 -1 journal do_read_entry(753664): bad header magic 2019-12-21T16:05:56.662 INFO:tasks.workunit.client.0.smithi172.stderr:119575: 2019-12-21T16:05:56.454+0000 7f1c9c0b8ec0 -1 osd.2 59 log_to_monitors {default=true} 2019-12-21T16:05:56.662 INFO:tasks.workunit.client.0.smithi172.stderr:119575: osd.2 up in weight 1 up_from 63 up_thru 49 down_at 60 last_clean_interval [15,59) [v2:127.0.0.1:6810/159433,v1:127.0.0.1:6811/159433] [v2:127.0.0.1:6812/159433,v1:127.0.0.1:6813/159433] exists,up f6c07742-f5aa-4b9f-b1b9-38cac5d974cc 2019-12-21T16:05:56.662 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=0 2019-12-21T16:05:56.662 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: break 2019-12-21T16:05:56.662 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: return 0 2019-12-21T16:05:56.663 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1239: objectstore_tool: wait_for_clean 2019-12-21T16:05:56.663 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1548: wait_for_clean: local cmd= 2019-12-21T16:05:56.663 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1549: wait_for_clean: local num_active_clean=-1 2019-12-21T16:05:56.663 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1550: wait_for_clean: local cur_active_clean 2019-12-21T16:05:56.663 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1551: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2019-12-21T16:05:56.663 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1551: wait_for_clean: get_timeout_delays 90 .1 2019-12-21T16:05:56.663 INFO:tasks.workunit.client.0.smithi172.stderr:119575: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1494: get_timeout_delays: shopt -q -o xtrace 2019-12-21T16:05:56.663 INFO:tasks.workunit.client.0.smithi172.stderr:119575: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1494: get_timeout_delays: echo true 2019-12-21T16:05:56.664 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1494: get_timeout_delays: local trace=true 2019-12-21T16:05:56.664 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1495: get_timeout_delays: true 2019-12-21T16:05:57.739 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1495: get_timeout_delays: shopt -u -o xtrace 2019-12-21T16:05:57.739 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1551: wait_for_clean: local -a delays 2019-12-21T16:05:57.739 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1552: wait_for_clean: local -i loop=0 2019-12-21T16:05:57.739 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1554: wait_for_clean: flush_pg_stats 2019-12-21T16:05:57.739 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1977: flush_pg_stats: local timeout=300 2019-12-21T16:05:57.740 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1979: flush_pg_stats: ceph osd ls 2019-12-21T16:05:57.740 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1979: flush_pg_stats: ids='0 2019-12-21T16:05:57.740 INFO:tasks.workunit.client.0.smithi172.stderr:119575: 1 2019-12-21T16:05:57.740 INFO:tasks.workunit.client.0.smithi172.stderr:119575: 2 2019-12-21T16:05:57.740 INFO:tasks.workunit.client.0.smithi172.stderr:119575: 3' 2019-12-21T16:05:57.740 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1980: flush_pg_stats: seqs= 2019-12-21T16:05:57.740 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: flush_pg_stats: for osd in $ids 2019-12-21T16:05:57.741 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2019-12-21T16:05:57.741 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: seq=21474836506 2019-12-21T16:05:57.741 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: flush_pg_stats: seqs=' 0-21474836506' 2019-12-21T16:05:57.741 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: flush_pg_stats: for osd in $ids 2019-12-21T16:05:57.741 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2019-12-21T16:05:57.741 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: seq=270582939650 2019-12-21T16:05:57.741 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: flush_pg_stats: seqs=' 0-21474836506 1-270582939650' 2019-12-21T16:05:57.742 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: flush_pg_stats: for osd in $ids 2019-12-21T16:05:57.742 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2019-12-21T16:05:57.742 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: seq=270582939650 2019-12-21T16:05:57.742 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: flush_pg_stats: seqs=' 0-21474836506 1-270582939650 2-270582939650' 2019-12-21T16:05:57.742 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: flush_pg_stats: for osd in $ids 2019-12-21T16:05:57.742 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2019-12-21T16:05:57.742 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: seq=249108103174 2019-12-21T16:05:57.743 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: flush_pg_stats: seqs=' 0-21474836506 1-270582939650 2-270582939650 3-249108103174' 2019-12-21T16:05:57.743 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: flush_pg_stats: for s in $seqs 2019-12-21T16:05:57.743 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: echo 0-21474836506 2019-12-21T16:05:57.743 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: cut -d - -f 1 2019-12-21T16:05:57.743 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: osd=0 2019-12-21T16:05:57.743 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: echo 0-21474836506 2019-12-21T16:05:57.743 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: cut -d - -f 2 2019-12-21T16:05:57.744 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: seq=21474836506 2019-12-21T16:05:57.744 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1989: flush_pg_stats: echo 'waiting osd.0 seq 21474836506' 2019-12-21T16:05:57.744 INFO:tasks.workunit.client.0.smithi172.stderr:119575: waiting osd.0 seq 21474836506 2019-12-21T16:05:57.744 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 0 2019-12-21T16:05:57.771 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 21474836505 -lt 21474lone/ceph-helpers.sh:871: activate_osd: '[' 1 = 1 ']' 2019-12-21T16:05:57.772 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph osd dump --format=json 2019-12-21T16:05:57.772 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: jq '.flags_set[]' 2019-12-21T16:05:57.772 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: grep -q '"noup"' 2019-12-21T16:05:57.772 INFO:tasks.workunit.client.0.smithi172.stderr:119575: 2019-12-21T16:05:56.062+0000 7f79ae029ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T16:05:57.772 INFO:tasks.workunit.client.0.smithi172.stderr:119575: 2019-12-21T16:05:56.064+0000 7f79ae029ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T16:05:57.772 INFO:tasks.workunit.client.0.smithi172.stderr:119575: 2019-12-21T16:05:56.065+0000 7f79ae029ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T16:05:57.772 INFO:tasks.workunit.client.0.smithi172.stderr:119575: 2019-12-21T16:05:56.067+0000 7f79ae029ec0 -1 Falling back to public interface 2019-12-21T16:05:57.773 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: wait_for_osd up 1 2019-12-21T16:05:57.773 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:914: wait_for_osd: local state=up 2019-12-21T16:05:57.773 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local id=1 2019-12-21T16:05:57.773 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: status=1 2019-12-21T16:05:57.773 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i=0 )) 2019-12-21T16:05:57.773 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i < 300 )) 2019-12-21T16:05:57.773 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: echo 0 2019-12-21T16:05:57.774 INFO:tasks.workunit.client.0.smithi172.stderr:119575: 0 2019-12-21T16:05:57.774 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: ceph osd dump 2019-12-21T16:05:57.774 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: grep 'osd.1 up' 2019-12-21T16:05:57.774 INFO:tasks.workunit.client.0.smithi172.stderr:119575: 2019-12-21T16:05:56.422+0000 7f79ae029ec0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2019-12-21T16:05:57.774 INFO:tasks.workunit.client.0.smithi172.stderr:119575: 2019-12-21T16:05:56.430+0000 7f79ae029ec0 -1 journal do_read_entry(823296): bad header magic 2019-12-21T16:05:57.774 INFO:tasks.workunit.client.0.smithi172.stderr:119575: 2019-12-21T16:05:56.430+0000 7f79ae029ec0 -1 journal do_read_entry(823296): bad header magic 2019-12-21T16:05:57.774 INFO:tasks.workunit.client.0.smithi172.stderr:119575: 2019-12-21T16:05:56.439+0000 7f79ae029ec0 -1 osd.1 59 log_to_monitors {default=true} 2019-12-21T16:05:57.775 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: sleep 1 2019-12-21T16:05:57.775 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i++ )) 2019-12-21T16:05:57.775 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i < 300 )) 2019-12-21T16:05:57.775 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: echo 1 2019-12-21T16:05:57.775 INFO:tasks.workunit.client.0.smithi172.stderr:119575: 1 2019-12-21T16:05:57.775 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: ceph osd dump 2019-12-21T16:05:57.775 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: grep 'osd.1 up' 2019-12-21T16:05:57.775 INFO:tasks.workunit.client.0.smithi172.stderr:119575: osd.1 up in weight 1 up_from 63 up_thru 63 down_at 60 last_clean_interval [52,59) [v2:127.0.0.1:6818/159457,v1:127.0.0.1:6819/159457] [v2:127.0.0.1:6820/159457,v1:127.0.0.1:6821/159457] exists,up 627d9f35-2c70-4ea0-8821-1ca4dc71496b 2019-12-21T16:05:57.776 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=0 2019-12-21T16:05:57.776 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: break 2019-12-21T16:05:57.776 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: return 0 2019-12-21T16:05:57.776 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1239: objectstore_tool: wait_for_clean 2019-12-21T16:05:57.776 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1548: wait_for_clean: local cmd= 2019-12-21T16:05:57.776 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1549: wait_for_clean: local num_active_clean=-1 2019-12-21T16:05:57.776 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1550: wait_for_clean: local cur_active_clean 2019-12-21T16:05:58.528 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1551: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2019-12-21T16:05:58.528 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1551: wait_for_clean: get_timeout_delays 90 .1 2019-12-21T16:05:58.528 INFO:tasks.workunit.client.0.smithi172.stderr:119575: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1494: get_timeout_delays: shopt -q -o xtrace 2019-12-21T16:05:58.528 INFO:tasks.workunit.client.0.smithi172.stderr:119575: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1494: get_timeout_delays: echo true 2019-12-21T16:05:58.528 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1494: get_timeout_delays: local trace=true 2019-12-21T16:05:58.529 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1495: get_timeout_delays: true 2019-12-21T16:05:58.529 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1495: get_timeout_delays: shopt -u -o xtrace 2019-12-21T16:05:58.529 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1551: wait_for_clean: local -a delays 2019-12-21T16:05:58.529 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1552: wait_for_clean: local -i loop=0 2019-12-21T16:05:58.529 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1554: wait_for_clean: flush_pg_stats 2019-12-21T16:05:58.529 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1977: flush_pg_stats: local timeout=300 2019-12-21T16:05:58.529 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1979: flush_pg_stats: ceph osd ls 2019-12-21T16:05:58.530 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1979: flush_pg_stats: ids='0 2019-12-21T16:05:58.530 INFO:tasks.workunit.client.0.smithi172.stderr:119575: 1 2019-12-21T16:05:58.530 INFO:tasks.workunit.client.0.smithi172.stderr:119575: 2 2019-12-21T16:05:58.530 INFO:tasks.workunit.client.0.smithi172.stderr:119575: 3' 2019-12-21T16:05:58.530 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1980: flush_pg_stats: seqs= 2019-12-21T16:05:58.530 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: flush_pg_stats: for osd in $ids 2019-12-21T16:05:58.530 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2019-12-21T16:05:58.531 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: seq=21474836507 2019-12-21T16:05:58.531 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: flush_pg_stats: seqs=' 0-21474836507' 2019-12-21T16:05:58.531 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: flush_pg_stats: for osd in $ids 2019-12-21T16:05:58.531 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2019-12-21T16:05:58.531 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: seq=270582939651 2019-12-21T16:05:58.531 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: flush_pg_stats: seqs=' 0-21474836507 1-270582939651' 2019-12-21T16:05:58.531 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: flush_pg_stats: for osd in $ids 2019-12-21T16:05:58.532 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2019-12-21T16:05:58.532 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: seq=270582939651 2019-12-21T16:05:58.532 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: flush_pg_stats: seqs=' 0-21474836507 1-270582939651 2-270582939651' 2019-12-21T16:05:58.532 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: flush_pg_stats: for osd in $ids 2019-12-21T16:05:58.532 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2019-12-21T16:05:58.532 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: seq=249108103175 2019-12-21T16:05:58.532 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: flush_pg_stats: seqs=' 0-21474836507 1-270582939651 2-270582939651 3-249108103175' 2019-12-21T16:05:58.533 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: flush_pg_stats: for s in $seqs 2019-12-21T16:05:58.533 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: echo 0-21474836507 2019-12-21T16:05:58.533 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: cut -d - -f 1 2019-12-21T16:05:58.533 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: osd=0 2019-12-21T16:06:00.494 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: echo 0-21474836507 2019-12-21T16:06:00.495 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: cut -d - -f 2 2019-12-21T16:06:00.495 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: seq=21474836507 2019-12-21T16:06:00.495 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1989: flush_pg_stats: echo 'waiting osd.0 seq 21474836507' 2019-12-21T16:06:00.495 INFO:tasks.workunit.client.0.smithi172.stderr:119575: waiting osd.0 seq 21474836507 2019-12-21T16:06:00.495 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 0 2019-12-21T16:06:00.495 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 21474836505 -lt 21474836507 2019-12-21T16:06:00.496 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1991: flush_pg_stats: sleep 1 2019-12-21T16:06:00.496 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1992: flush_pg_stats: '[' 300 -eq 0 ']' 2019-12-21T16:06:00.496 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 0 2019-12-21T16:06:00.496 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 21474836508 -lt 21474836507 2019-12-21T16:06:00.496 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: flush_pg_stats: for s in $seqs 2019-12-21T16:06:00.496 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: echo 1-270582939651 2019-12-21T16:06:00.497 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: cut -d - -f 1 2019-12-21T16:06:00.497 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: osd=1 2019-12-21T16:06:00.497 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: echo 1-270582939651 2019-12-21T16:06:00.497 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: cut -d - -f 2 2019-12-21T16:06:00.497 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: seq=270582939651 2019-12-21T16:06:00.497 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1989: flush_pg_stats: echo 'waiting osd.1 seq 270582939651' 2019-12-21T16:06:00.497 INFO:tasks.workunit.client.0.smithi172.stderr:119575: waiting osd.1 seq 270582939651 2019-12-21T16:06:00.498 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 1 2019-12-21T16:06:00.498 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 270582939651 -lt 270582939651 2019-12-21T16:06:00.498 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: flush_pg_stats: for s in $seqs 2019-12-21T16:06:00.498 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: echo 2-270582939651 2019-12-21T16:06:00.498 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: cut -d - -f 1 2019-12-21T16:06:00.498 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: osd=2 2019-12-21T16:06:00.498 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: echo 2-270582939651 2019-12-21T16:06:00.498 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: cut -d - -f 2 2019-12-21T16:06:00.499 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: seq=270582939651 2019-12-21T16:06:00.499 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1989: flush_pg_stats: echo 'waiting osd.2 seq 270582939651' 2019-12-21T16:06:00.499 INFO:tasks.workunit.client.0.smithi172.stderr:119575: waiting osd.2 seq 270582939651 2019-12-21T16:06:00.499 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 2 2019-12-21T16:06:00.499 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 270582939651 -lt 270582939651 2019-12-21T16:06:00.499 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: flush_pg_stats: for s in $seqs 2019-12-21T16:06:00.499 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: echo 3-249108103175 2019-12-21T16:06:00.500 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: cut -d - -f 1 2019-12-21T16:06:00.500 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: osd=3 2019-12-21T16:06:00.734 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalon836506 2019-12-21T16:06:00.734 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1991: flush_pg_stats: sleep 1 2019-12-21T16:06:00.734 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1992: flush_pg_stats: '[' 300 -eq 0 ']' 2019-12-21T16:06:00.735 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 0 2019-12-21T16:06:00.735 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 21474836505 -lt 21474836506 2019-12-21T16:06:00.735 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1991: flush_pg_stats: sleep 1 2019-12-21T16:06:00.735 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1992: flush_pg_stats: '[' 299 -eq 0 ']' 2019-12-21T16:06:00.735 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 0 2019-12-21T16:06:00.735 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 21474836508 -lt 21474836506 2019-12-21T16:06:00.735 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: flush_pg_stats: for s in $seqs 2019-12-21T16:06:00.736 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: echo 1-270582939650 2019-12-21T16:06:00.736 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: cut -d - -f 1 2019-12-21T16:06:00.736 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: osd=1 2019-12-21T16:06:00.736 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: echo 1-270582939650 2019-12-21T16:06:00.736 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: cut -d - -f 2 2019-12-21T16:06:00.736 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: seq=270582939650 2019-12-21T16:06:00.737 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1989: flush_pg_stats: echo 'waiting osd.1 seq 270582939650' 2019-12-21T16:06:00.737 INFO:tasks.workunit.client.0.smithi172.stderr:119575: waiting osd.1 seq 270582939650 2019-12-21T16:06:00.737 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 1 2019-12-21T16:06:00.737 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 270582939651 -lt 270582939650 2019-12-21T16:06:00.737 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: flush_pg_stats: for s in $seqs 2019-12-21T16:06:00.737 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: echo 2-270582939650 2019-12-21T16:06:00.737 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: cut -d - -f 1 2019-12-21T16:06:00.737 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: osd=2 2019-12-21T16:06:00.738 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: echo 2-270582939650 2019-12-21T16:06:00.738 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: cut -d - -f 2 2019-12-21T16:06:00.738 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: seq=270582939650 2019-12-21T16:06:00.738 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1989: flush_pg_stats: echo 'waiting osd.2 seq 270582939650' 2019-12-21T16:06:00.738 INFO:tasks.workunit.client.0.smithi172.stderr:119575: waiting osd.2 seq 270582939650 2019-12-21T16:06:00.738 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 2 2019-12-21T16:06:00.738 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 270582939651 -lt 270582939650 2019-12-21T16:06:00.739 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: flush_pg_stats: for s in $seqs 2019-12-21T16:06:00.739 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: echo 3-249108103174 2019-12-21T16:06:00.739 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: cut -d - -f 1 2019-12-21T16:06:00.739 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: osd=3 2019-12-21T16:06:00.739 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: echo 3-249108103174 2019-12-21T16:06:01.520 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: cut -d - -f e/ceph-helpers.sh:1988: flush_pg_stats: echo 3-249108103175 2019-12-21T16:06:01.521 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: cut -d - -f 2 2019-12-21T16:06:01.521 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: seq=249108103175 2019-12-21T16:06:01.521 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1989: flush_pg_stats: echo 'waiting osd.3 seq 249108103175' 2019-12-21T16:06:01.521 INFO:tasks.workunit.client.0.smithi172.stderr:119575: waiting osd.3 seq 249108103175 2019-12-21T16:06:01.521 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 3 2019-12-21T16:06:01.521 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 249108103175 -lt 249108103175 2019-12-21T16:06:01.522 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1555: wait_for_clean: get_num_pgs 2019-12-21T16:06:01.522 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: ceph --format json status 2019-12-21T16:06:01.522 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: jq .pgmap.num_pgs 2019-12-21T16:06:01.522 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1555: wait_for_clean: test 5 == 0 2019-12-21T16:06:01.522 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1559: wait_for_clean: true 2019-12-21T16:06:01.522 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1563: wait_for_clean: get_num_active_clean 2019-12-21T16:06:01.522 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: get_num_active_clean: local expression 2019-12-21T16:06:01.523 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2019-12-21T16:06:01.523 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("stale") | not)' 2019-12-21T16:06:01.523 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: ceph --format json pg dump pgs 2019-12-21T16:06:01.523 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2019-12-21T16:06:01.523 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1563: wait_for_clean: cur_active_clean=5 2019-12-21T16:06:01.523 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1564: wait_for_clean: get_num_pgs 2019-12-21T16:06:01.523 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: ceph --format json status 2019-12-21T16:06:01.524 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: jq .pgmap.num_pgs 2019-12-21T16:06:01.524 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1564: wait_for_clean: test 5 = 5 2019-12-21T16:06:01.524 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1564: wait_for_clean: break 2019-12-21T16:06:01.524 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: return 0 2019-12-21T16:06:01.524 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1911: run_in_background: return 0 2019-12-21T16:06:01.524 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1946: wait_background: for pid in $pids 2019-12-21T16:06:01.524 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1947: wait_background: wait 159293 2019-12-21T16:06:01.831 INFO:tasks.workunit.client.0.smithi172.stderr:2 2019-12-21T16:06:01.831 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: seq=249108103174 2019-12-21T16:06:01.832 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1989: flush_pg_stats: echo 'waiting osd.3 seq 249108103174' 2019-12-21T16:06:01.832 INFO:tasks.workunit.client.0.smithi172.stderr:119575: waiting osd.3 seq 249108103174 2019-12-21T16:06:01.832 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 3 2019-12-21T16:06:01.832 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 249108103175 -lt 249108103174 2019-12-21T16:06:01.832 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1555: wait_for_clean: get_num_pgs 2019-12-21T16:06:01.832 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: ceph --format json status 2019-12-21T16:06:01.833 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: jq .pgmap.num_pgs 2019-12-21T16:06:01.833 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1555: wait_for_clean: test 5 == 0 2019-12-21T16:06:01.833 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1559: wait_for_clean: true 2019-12-21T16:06:01.833 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1563: wait_for_clean: get_num_active_clean 2019-12-21T16:06:01.833 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: get_num_active_clean: local expression 2019-12-21T16:06:01.833 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2019-12-21T16:06:01.833 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("stale") | not)' 2019-12-21T16:06:01.834 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: ceph --format json pg dump pgs 2019-12-21T16:06:01.834 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2019-12-21T16:06:01.834 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1563: wait_for_clean: cur_active_clean=5 2019-12-21T16:06:01.834 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1564: wait_for_clean: get_num_pgs 2019-12-21T16:06:01.834 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: ceph --format json status 2019-12-21T16:06:01.834 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: jq .pgmap.num_pgs 2019-12-21T16:06:01.834 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1564: wait_for_clean: test 5 = 5 2019-12-21T16:06:01.835 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1564: wait_for_clean: break 2019-12-21T16:06:01.835 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: return 0 2019-12-21T16:06:01.835 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1911: run_in_background: return 0 2019-12-21T16:06:01.835 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1954: wait_background: eval 'pids='\'''\''' 2019-12-21T16:06:01.835 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1954: wait_background: pids= 2019-12-21T16:06:01.835 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1956: wait_background: return 0 2019-12-21T16:06:01.835 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: return_code=0 2019-12-21T16:06:01.836 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:204: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2019-12-21T16:06:01.836 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:209: corrupt_and_repair_two: get_pg ecpool SOMETHING 2019-12-21T16:06:01.836 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1029: get_pg: local poolname=ecpool 2019-12-21T16:06:01.836 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1030: get_pg: local objectname=SOMETHING 2019-12-21T16:06:01.836 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1032: get_pg: ceph --format json osd map ecpool SOMETHING 2019-12-21T16:06:01.836 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1032: get_pg: jq -r .pgid 2019-12-21T16:06:02.063 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:209: corrupt_and_repair_two: local pg=2.0 2019-12-21T16:06:02.063 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:210: corrupt_and_repair_two: repair 2.0 2019-12-21T16:06:02.063 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: repair: local pgid=2.0 2019-12-21T16:06:02.064 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1665: repair: get_last_scrub_stamp 2.0 2019-12-21T16:06:02.064 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=2.0 2019-12-21T16:06:02.064 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T16:06:02.064 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T16:06:02.064 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2019-12-21T16:06:02.286 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1665: repair: local last_scrub=2019-12-21T16:05:35.219372+0000 2019-12-21T16:06:02.287 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1666: repair: ceph pg repair 2.0 2019-12-21T16:06:02.502 INFO:tasks.workunit.client.0.smithi172.stderr:instructing pg 2.0s0 on osd.3 to repair 2019-12-21T16:06:02.510 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: repair: wait_for_scrub 2.0 2019-12-21T16:05:35.219372+0000 2019-12-21T16:06:02.511 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1789: wait_for_scrub: local pgid=2.0 2019-12-21T16:06:02.511 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1790: wait_for_scrub: local last_scrub=2019-12-21T16:05:35.219372+0000 2019-12-21T16:06:02.511 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1791: wait_for_scrub: local sname=last_scrub_stamp 2019-12-21T16:06:02.511 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i=0 )) 2019-12-21T16:06:02.511 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T16:06:02.511 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2019-12-21T16:06:02.512 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=2.0 2019-12-21T16:06:02.512 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T16:06:02.512 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T16:06:02.512 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2019-12-21T16:06:02.732 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T16:05:35.219372+0000 '>' 2019-12-21T16:05:35.219372+0000 2019-12-21T16:06:02.732 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T16:06:03.733 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T16:06:03.733 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T16:06:03.734 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2019-12-21T16:06:03.734 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=2.0 2019-12-21T16:06:03.734 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T16:06:03.734 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T16:06:03.734 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2019-12-21T16:06:03.969 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T16:05:35.219372+0000 '>' 2019-12-21T16:05:35.219372+0000 2019-12-21T16:06:03.969 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T16:06:04.970 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T16:06:04.970 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T16:06:04.971 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2019-12-21T16:06:04.971 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=2.0 2019-12-21T16:06:04.971 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T16:06:04.972 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T16:06:04.972 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2019-12-21T16:06:05.230 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T16:06:04.405865+0000 '>' 2019-12-21T16:05:35.219372+0000 2019-12-21T16:06:05.231 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1795: wait_for_scrub: return 0 2019-12-21T16:06:05.231 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:214: corrupt_and_repair_two: pids= 2019-12-21T16:06:05.231 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:215: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2019-12-21T16:06:05.231 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1907: run_in_background: local pid_variable=pids 2019-12-21T16:06:05.231 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1908: run_in_background: shift 2019-12-21T16:06:05.231 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1912: run_in_background: eval 'pids+=" 160841"' 2019-12-21T16:06:05.232 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1912: run_in_background: pids+=' 160841' 2019-12-21T16:06:05.232 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:216: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 2 SOMETHING list-attrs 2019-12-21T16:06:05.232 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1907: run_in_background: local pid_variable=pids 2019-12-21T16:06:05.232 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1908: run_in_background: shift 2019-12-21T16:06:05.232 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1911: run_in_background: objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2019-12-21T16:06:05.232 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1912: run_in_background: eval 'pids+=" 160843"' 2019-12-21T16:06:05.233 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1912: run_in_background: pids+=' 160843' 2019-12-21T16:06:05.233 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:217: corrupt_and_repair_two: wait_background pids 2019-12-21T16:06:05.233 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1943: wait_background: pids=' 160841 160843' 2019-12-21T16:06:05.233 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1945: wait_background: return_code=0 2019-12-21T16:06:05.233 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1946: wait_background: for pid in $pids 2019-12-21T16:06:05.233 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1911: run_in_background: sed 's/^/119575: /' 2019-12-21T16:06:05.234 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1947: wait_background: wait 160841 2019-12-21T16:06:05.234 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1911: run_in_background: objectstore_tool td/osd-scrub-repair 2 SOMETHING list-attrs 2019-12-21T16:06:05.234 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1911: run_in_background: sed 's/^/119575: /' 2019-12-21T16:06:06.313 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: objectstore_tool: local dir=td/osd-scrub-repair 2019-12-21T16:06:06.313 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: objectstore_tool: shift 2019-12-21T16:06:06.313 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1235: objectstore_tool: local id=1 2019-12-21T16:06:06.314 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1236: objectstore_tool: shift 2019-12-21T16:06:06.314 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1238: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING list-attrs 2019-12-21T16:06:06.314 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1205: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2019-12-21T16:06:06.314 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nowait: shift 2019-12-21T16:06:06.314 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1207: _objectstore_tool_nowait: local id=1 2019-12-21T16:06:06.314 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1208: _objectstore_tool_nowait: shift 2019-12-21T16:06:06.314 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1210: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2019-12-21T16:06:06.315 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2019-12-21T16:06:06.315 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2019-12-21T16:06:06.315 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2019-12-21T16:06:06.315 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2019-12-21T16:06:06.315 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2019-12-21T16:06:06.315 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2019-12-21T16:06:06.316 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1212: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING list-attrs 2019-12-21T16:06:06.316 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1193: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2019-12-21T16:06:06.316 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1194: _objectstore_tool_nodown: shift 2019-12-21T16:06:06.316 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1195: _objectstore_tool_nodown: local id=1 2019-12-21T16:06:06.316 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: _objectstore_tool_nodown: shift 2019-12-21T16:06:06.316 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2019-12-21T16:06:06.316 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING list-attrs 2019-12-21T16:06:06.317 INFO:tasks.workunit.client.0.smithi172.stderr:119575: _ 2019-12-21T16:06:06.317 INFO:tasks.workunit.client.0.smithi172.stderr:119575: hinfo_key 2019-12-21T16:06:06.317 INFO:tasks.workunit.client.0.smithi172.stderr:119575: snapset 2019-12-21T16:06:06.317 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1213: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2019-12-21T16:06:06.317 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:842: activate_osd: local dir=td/osd-scrub-repair 2019-12-21T16:06:06.317 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: shift 2019-12-21T16:06:06.317 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: local id=1 2019-12-21T16:06:06.318 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: shift 2019-12-21T16:06:06.319 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local osd_data=td/osd-scrub-repair/1 2019-12-21T16:06:06.320 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local 'ceph_args=--fsid=973e3fc1-e63e-4bde-9db6-84c68fd5e93e --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2019-12-21T16:06:06.320 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2019-12-21T16:06:06.320 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helper119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: objectstore_tool: local dir=td/osd-scrub-repair 2019-12-21T16:06:06.320 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: objectstore_tool: shift 2019-12-21T16:06:06.321 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1235: objectstore_tool: local id=2 2019-12-21T16:06:06.321 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1236: objectstore_tool: shift 2019-12-21T16:06:06.321 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1238: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 2 SOMETHING list-attrs 2019-12-21T16:06:06.321 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1205: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2019-12-21T16:06:06.322 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nowait: shift 2019-12-21T16:06:06.322 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1207: _objectstore_tool_nowait: local id=2 2019-12-21T16:06:06.322 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1208: _objectstore_tool_nowait: shift 2019-12-21T16:06:06.322 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1210: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.2 2019-12-21T16:06:06.322 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2019-12-21T16:06:06.322 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2019-12-21T16:06:06.323 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2019-12-21T16:06:06.323 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2019-12-21T16:06:06.324 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2019-12-21T16:06:06.324 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2019-12-21T16:06:06.324 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1212: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 2 SOMETHING list-attrs 2019-12-21T16:06:06.324 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1193: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2019-12-21T16:06:06.324 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1194: _objectstore_tool_nodown: shift 2019-12-21T16:06:06.325 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1195: _objectstore_tool_nodown: local id=2 2019-12-21T16:06:06.325 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: _objectstore_tool_nodown: shift 2019-12-21T16:06:06.325 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/2 2019-12-21T16:06:06.325 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 SOMETHING list-attrs 2019-12-21T16:06:06.325 INFO:tasks.workunit.client.0.smithi172.stderr:119575: _ 2019-12-21T16:06:06.325 INFO:tasks.workunit.client.0.smithi172.stderr:119575: hinfo_key 2019-12-21T16:06:06.325 INFO:tasks.workunit.client.0.smithi172.stderr:119575: snapset 2019-12-21T16:06:06.326 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1213: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 2 2019-12-21T16:06:06.326 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:842: activate_osd: local dir=td/osd-scrub-repair 2019-12-21T16:06:06.326 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: shift 2019-12-21T16:06:06.326 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: local id=2 2019-12-21T16:06:06.326 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: shift 2019-12-21T16:06:06.326 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local osd_data=td/osd-scrub-repair/2 2019-12-21T16:06:06.326 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local 'ceph_args=--fsid=973e3fc1-e63e-4bde-9db6-84c68fd5e93e --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2019-12-21T16:06:06.327 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2019-12-21T16:06:06.327 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-journal-size=100' 2019-12-21T16:06:06.327 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2019-12-21T16:06:06.328 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2019-12-21T16:06:06.328 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2019-12-21T16:06:06.328 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --chdir=' 2019-12-21T16:06:06.328 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+= 2019-12-21T16:06:06.328 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2019-12-21T16:06:06.328 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: get_asok_path 2019-12-21T16:06:06.329 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2019-12-21T16:06:06.329 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2019-12-21T16:06:06.329 INFO:tasks.workunit.client.0.smithi172.stderr:119575: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2019-12-21T16:06:06.329 INFO:tasks.workunit.client.0.smithi172.stderr:119575: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T16:06:06.329 INFO:tasks.workunit.client.0.smithi172.stderr:119575: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.119575 2019-12-21T16:06:06.329 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.119575/$cluster-$name.asok' 2019-12-21T16:06:06.330 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.119575/$cluster-$name.asok' 2019-12-21T16:06:06.330 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --debug-osd=20' 2019-12-21T16:06:06.330 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2019-12-21T16:06:06.330 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2019-12-21T16:06:06.330 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2019-12-21T16:06:06.330 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2019-12-21T16:06:06.330 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2019-12-21T16:06:06.331 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' ' 2019-12-21T16:06:06.331 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+= 2019-12-21T16:06:06.331 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: mkdir -p td/osd-scrub-repair/1 2019-12-21T16:06:06.331 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: echo start osd.1 2019-12-21T16:06:06.331 INFO:tasks.workunit.client.0.smithi172.stderr:119575: start osd.1 2019-12-21T16:06:06.332 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph-osd -i 1 --fsid=973e3fc1-e63e-4bde-9db6-84c68fd5e93e --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.119575/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2019-12-21T16:06:06.332 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: cat td/osd-scrub-repair/1/whoami 2019-12-21T16:06:06.332 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0s.sh:850: activate_osd: ceph_args+=' --osd-journal-size=100' 2019-12-21T16:06:06.332 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2019-12-21T16:06:06.332 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2019-12-21T16:06:06.332 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2019-12-21T16:06:06.332 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --chdir=' 2019-12-21T16:06:06.333 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+= 2019-12-21T16:06:06.333 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2019-12-21T16:06:06.333 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: get_asok_path 2019-12-21T16:06:06.333 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2019-12-21T16:06:06.333 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2019-12-21T16:06:06.333 INFO:tasks.workunit.client.0.smithi172.stderr:119575: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2019-12-21T16:06:06.333 INFO:tasks.workunit.client.0.smithi172.stderr:119575: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T16:06:06.334 INFO:tasks.workunit.client.0.smithi172.stderr:119575: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.119575 2019-12-21T16:06:06.334 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.119575/$cluster-$name.asok' 2019-12-21T16:06:06.334 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.119575/$cluster-$name.asok' 2019-12-21T16:06:06.334 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --debug-osd=20' 2019-12-21T16:06:06.334 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2019-12-21T16:06:06.334 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2019-12-21T16:06:06.334 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2019-12-21T16:06:06.335 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2019-12-21T16:06:06.335 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2019-12-21T16:06:06.335 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' ' 2019-12-21T16:06:06.335 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+= 2019-12-21T16:06:06.335 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: mkdir -p td/osd-scrub-repair/2 2019-12-21T16:06:06.335 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: echo start osd.2 2019-12-21T16:06:06.335 INFO:tasks.workunit.client.0.smithi172.stderr:119575: start osd.2 2019-12-21T16:06:06.336 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph-osd -i 2 --fsid=973e3fc1-e63e-4bde-9db6-84c68fd5e93e --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.119575/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2019-12-21T16:06:06.336 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: cat td/osd-scrub-repair/2/whoami 2019-12-21T16:06:08.231 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: '[' 1 = 1 ']' 2019-12-21T16:06:08.232 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph osd dump --format=json 2019-12-21T16:06:08.232 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: jq '.flags_set[]' 2019-12-21T16:06:08.232 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: grep -q '"noup"' 2019-12-21T16:06:08.232 INFO:tasks.workunit.client.0.smithi172.stderr:119575: 2019-12-21T16:06:06.335+0000 7f2706822ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T16:06:08.232 INFO:tasks.workunit.client.0.smithi172.stderr:119575: 2019-12-21T16:06:06.337+0000 7f2706822ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T16:06:08.232 INFO:tasks.workunit.client.0.smithi172.stderr:119575: 2019-12-21T16:06:06.339+0000 7f2706822ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T16:06:08.233 INFO:tasks.workunit.client.0.smithi172.stderr:119575: 2019-12-21T16:06:06.341+0000 7f2706822ec0 -1 Falling back to public interface 2019-12-21T16:06:08.233 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: wait_for_osd up 1 2019-12-21T16:06:08.233 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:914: wait_for_osd: local state=up 2019-12-21T16:06:08.233 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local id=1 2019-12-21T16:06:08.233 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: status=1 2019-12-21T16:06:08.233 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i=0 )) 2019-12-21T16:06:08.234 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i < 300 )) 2019-12-21T16:06:08.234 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: echo 0 2019-12-21T16:06:08.234 INFO:tasks.workunit.client.0.smithi172.stderr:119575: 0 2019-12-21T16:06:08.234 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: ceph osd dump 2019-12-21T16:06:08.234 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: grep 'osd.1 up' 2019-12-21T16:06:08.234 INFO:tasks.workunit.client.0.smithi172.stderr:119575: 2019-12-21T16:06:06.938+0000 7f2706822ec0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2019-12-21T16:06:08.234 INFO:tasks.workunit.client.0.smithi172.stderr:119575: 2019-12-21T16:06:06.946+0000 7f2706822ec0 -1 journal do_read_entry(966656): bad header magic 2019-12-21T16:06:08.235 INFO:tasks.workunit.client.0.smithi172.stderr:119575: 2019-12-21T16:06:06.946+0000 7f2706822ec0 -1 journal do_read_entry(966656): bad header magic 2019-12-21T16:06:08.235 INFO:tasks.workunit.client.0.smithi172.stderr:119575: 2019-12-21T16:06:06.954+0000 7f2706822ec0 -1 osd.1 64 log_to_monitors {default=true} 2019-12-21T16:06:08.235 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: sleep 1 2019-12-21T16:06:08.235 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i++ )) 2019-12-21T16:06:08.235 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i < 300 )) 2019-12-21T16:06:08.235 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: echo 1 2019-12-21T16:06:08.236 INFO:tasks.workunit.client.0.smithi172.stderr:119575: 1 2019-12-21T16:06:08.236 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: ceph osd dump 2019-12-21T16:06:08.236 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: grep 'osd.1 up' 2019-12-21T16:06:08.236 INFO:tasks.workunit.client.0.smithi172.stderr:119575: osd.1 up in weight 1 up_from 68 up_thru 68 down_at 65 last_clean_interval [63,64) [v2:127.0.0.1:6815/160992,v1:127.0.0.1:6817/160992] [v2:127.0.0.1:6819/160992,v1:127.0.0.1:6821/160992] exists,up 627d9f35-2c70-4ea0-8821-1ca4dc71496b 2019-12-21T16:06:08.236 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=0 2019-12-21T16:06:08.236 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: break 2019-12-21T16:06:08.236 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: return 0 2019-12-21T16:06:08.237 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1239: objectstore_tool: wait_for_clean 2019-12-21T16:06:08.237 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1548: wait_for_clean: local cmd= 2019-12-21T16:06:08.237 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1549: wait_for_clean: local num_active_clean=-1 2019-12-21T16:06:08.237 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1550: wait_for_clean: local cur_active_clean 2019-12-21T16:06:08.238 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1551: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CL/qa/standalone/ceph-helpers.sh:871: activate_osd: '[' 2 = 2 ']' 2019-12-21T16:06:08.238 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph osd dump --format=json 2019-12-21T16:06:08.238 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: grep -q '"noup"' 2019-12-21T16:06:08.238 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: jq '.flags_set[]' 2019-12-21T16:06:08.238 INFO:tasks.workunit.client.0.smithi172.stderr:119575: 2019-12-21T16:06:06.334+0000 7f242fc40ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T16:06:08.238 INFO:tasks.workunit.client.0.smithi172.stderr:119575: 2019-12-21T16:06:06.337+0000 7f242fc40ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T16:06:08.239 INFO:tasks.workunit.client.0.smithi172.stderr:119575: 2019-12-21T16:06:06.338+0000 7f242fc40ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T16:06:08.239 INFO:tasks.workunit.client.0.smithi172.stderr:119575: 2019-12-21T16:06:06.340+0000 7f242fc40ec0 -1 Falling back to public interface 2019-12-21T16:06:08.239 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: wait_for_osd up 2 2019-12-21T16:06:08.239 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:914: wait_for_osd: local state=up 2019-12-21T16:06:08.239 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local id=2 2019-12-21T16:06:08.239 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: status=1 2019-12-21T16:06:08.239 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i=0 )) 2019-12-21T16:06:08.240 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i < 300 )) 2019-12-21T16:06:08.240 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: echo 0 2019-12-21T16:06:08.240 INFO:tasks.workunit.client.0.smithi172.stderr:119575: 0 2019-12-21T16:06:08.240 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: ceph osd dump 2019-12-21T16:06:08.240 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: grep 'osd.2 up' 2019-12-21T16:06:08.240 INFO:tasks.workunit.client.0.smithi172.stderr:119575: 2019-12-21T16:06:06.938+0000 7f242fc40ec0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2019-12-21T16:06:08.240 INFO:tasks.workunit.client.0.smithi172.stderr:119575: 2019-12-21T16:06:06.946+0000 7f242fc40ec0 -1 journal do_read_entry(892928): bad header magic 2019-12-21T16:06:08.241 INFO:tasks.workunit.client.0.smithi172.stderr:119575: 2019-12-21T16:06:06.946+0000 7f242fc40ec0 -1 journal do_read_entry(892928): bad header magic 2019-12-21T16:06:08.241 INFO:tasks.workunit.client.0.smithi172.stderr:119575: 2019-12-21T16:06:06.954+0000 7f242fc40ec0 -1 osd.2 64 log_to_monitors {default=true} 2019-12-21T16:06:08.241 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: sleep 1 2019-12-21T16:06:08.241 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i++ )) 2019-12-21T16:06:08.241 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i < 300 )) 2019-12-21T16:06:08.241 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: echo 1 2019-12-21T16:06:08.241 INFO:tasks.workunit.client.0.smithi172.stderr:119575: 1 2019-12-21T16:06:08.241 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: ceph osd dump 2019-12-21T16:06:08.242 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: grep 'osd.2 up' 2019-12-21T16:06:08.242 INFO:tasks.workunit.client.0.smithi172.stderr:119575: osd.2 up in weight 1 up_from 68 up_thru 49 down_at 65 last_clean_interval [63,64) [v2:127.0.0.1:6810/160991,v1:127.0.0.1:6811/160991] [v2:127.0.0.1:6812/160991,v1:127.0.0.1:6813/160991] exists,up f6c07742-f5aa-4b9f-b1b9-38cac5d974cc 2019-12-21T16:06:08.242 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=0 2019-12-21T16:06:08.242 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: break 2019-12-21T16:06:08.242 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: return 0 2019-12-21T16:06:08.242 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1239: objectstore_tool: wait_for_clean 2019-12-21T16:06:08.243 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1548: wait_for_clean: local cmd= 2019-12-21T16:06:08.243 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1549: wait_for_clean: local num_active_clean=-1 2019-12-21T16:06:08.243 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1550: wait_for_clean: local cur_active_clean 2019-12-21T16:06:09.037 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1551: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2019-12-21T16:06:09.037 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1551: wait_for_clean: get_timeout_delays 90 .1 2019-12-21T16:06:09.037 INFO:tasks.workunit.client.0.smithi172.stderr:119575: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1494: get_timeout_delays: shopt -q -o xtrace 2019-12-21T16:06:09.037 INFO:tasks.workunit.client.0.smithi172.stderr:119575: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1494: get_timeout_delays: echo true 2019-12-21T16:06:09.038 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1494: get_timeout_delays: local trace=true 2019-12-21T16:06:09.038 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1495: get_timeout_delays: true 2019-12-21T16:06:09.038 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1495: get_timeout_delays: shopt -u -o xtrace 2019-12-21T16:06:09.038 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1551: wait_for_clean: local -a delays 2019-12-21T16:06:09.038 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1552: wait_for_clean: local -i loop=0 2019-12-21T16:06:09.038 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1554: wait_for_clean: flush_pg_stats 2019-12-21T16:06:09.038 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1977: flush_pg_stats: local timeout=300 2019-12-21T16:06:09.039 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1979: flush_pg_stats: ceph osd ls 2019-12-21T16:06:09.039 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1979: flush_pg_stats: ids='0 2019-12-21T16:06:09.040 INFO:tasks.workunit.client.0.smithi172.stderr:119575: 1 2019-12-21T16:06:09.040 INFO:tasks.workunit.client.0.smithi172.stderr:119575: 2 2019-12-21T16:06:09.041 INFO:tasks.workunit.client.0.smithi172.stderr:119575: 3' 2019-12-21T16:06:09.041 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1980: flush_pg_stats: seqs= 2019-12-21T16:06:09.041 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: flush_pg_stats: for osd in $ids 2019-12-21T16:06:09.041 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2019-12-21T16:06:09.041 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: seq=21474836511 2019-12-21T16:06:09.041 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: flush_pg_stats: seqs=' 0-21474836511' 2019-12-21T16:06:09.041 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: flush_pg_stats: for osd in $ids 2019-12-21T16:06:09.042 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2019-12-21T16:06:09.042 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: seq=292057776131 2019-12-21T16:06:09.042 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: flush_pg_stats: seqs=' 0-21474836511 1-292057776131' 2019-12-21T16:06:09.042 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: flush_pg_stats: for osd in $ids 2019-12-21T16:06:09.042 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2019-12-21T16:06:09.042 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: seq=292057776131 2019-12-21T16:06:09.043 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: flush_pg_stats: seqs=' 0-21474836511 1-292057776131 2-292057776131' 2019-12-21T16:06:09.043 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: flush_pg_stats: for osd in $ids 2019-12-21T16:06:09.043 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2019-12-21T16:06:09.043 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: seq=249108103178 2019-12-21T16:06:09.043 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: flush_pg_stats: seqs=' 0-21474836511 1-292057776131 2-292057776131 3-249108103178' 2019-12-21T16:06:09.043 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: flush_pg_stats: for s in $seqs 2019-12-21T16:06:09.043 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: echo 0-21474836511 2019-12-21T16:06:09.044 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: cut -d - -f 1 2019-12-21T16:06:09.044 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: osd=0 2019-12-21T16:06:09.044 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: echo 0-2EAN_TIMEOUT .1)) 2019-12-21T16:06:09.044 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1551: wait_for_clean: get_timeout_delays 90 .1 2019-12-21T16:06:09.044 INFO:tasks.workunit.client.0.smithi172.stderr:119575: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1494: get_timeout_delays: shopt -q -o xtrace 2019-12-21T16:06:09.044 INFO:tasks.workunit.client.0.smithi172.stderr:119575: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1494: get_timeout_delays: echo true 2019-12-21T16:06:09.044 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1494: get_timeout_delays: local trace=true 2019-12-21T16:06:09.045 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1495: get_timeout_delays: true 2019-12-21T16:06:09.045 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1495: get_timeout_delays: shopt -u -o xtrace 2019-12-21T16:06:09.045 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1551: wait_for_clean: local -a delays 2019-12-21T16:06:09.045 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1552: wait_for_clean: local -i loop=0 2019-12-21T16:06:09.045 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1554: wait_for_clean: flush_pg_stats 2019-12-21T16:06:09.045 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1977: flush_pg_stats: local timeout=300 2019-12-21T16:06:09.045 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1979: flush_pg_stats: ceph osd ls 2019-12-21T16:06:09.046 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1979: flush_pg_stats: ids='0 2019-12-21T16:06:09.046 INFO:tasks.workunit.client.0.smithi172.stderr:119575: 1 2019-12-21T16:06:09.046 INFO:tasks.workunit.client.0.smithi172.stderr:119575: 2 2019-12-21T16:06:09.046 INFO:tasks.workunit.client.0.smithi172.stderr:119575: 3' 2019-12-21T16:06:09.046 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1980: flush_pg_stats: seqs= 2019-12-21T16:06:09.046 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: flush_pg_stats: for osd in $ids 2019-12-21T16:06:09.047 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2019-12-21T16:06:09.047 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: seq=21474836512 2019-12-21T16:06:09.047 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: flush_pg_stats: seqs=' 0-21474836512' 2019-12-21T16:06:09.047 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: flush_pg_stats: for osd in $ids 2019-12-21T16:06:09.048 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2019-12-21T16:06:09.048 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: seq=292057776130 2019-12-21T16:06:09.048 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: flush_pg_stats: seqs=' 0-21474836512 1-292057776130' 2019-12-21T16:06:09.049 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: flush_pg_stats: for osd in $ids 2019-12-21T16:06:09.049 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2019-12-21T16:06:09.049 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: seq=292057776130 2019-12-21T16:06:09.049 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: flush_pg_stats: seqs=' 0-21474836512 1-292057776130 2-292057776130' 2019-12-21T16:06:09.049 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: flush_pg_stats: for osd in $ids 2019-12-21T16:06:09.050 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2019-12-21T16:06:09.050 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: seq=249108103179 2019-12-21T16:06:09.050 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: flush_pg_stats: seqs=' 0-21474836512 1-292057776130 2-292057776130 3-249108103179' 2019-12-21T16:06:09.050 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: flush_pg_stats: for s in $seqs 2019-12-21T16:06:09.050 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: echo 0-21474836512 2019-12-21T16:06:09.050 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: cut -d - -f 1 2019-12-21T16:06:09.050 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: osd=0 2019-12-21T16:06:09.777 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: echo 0-21474836511 2019-12-21T16:06:09.777 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: cut -d - -f 2 2019-12-21T16:06:09.777 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: seq=21474836511 2019-12-21T16:06:09.778 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1989: flush_pg_stats: echo 'waiting osd.0 seq 21474836511' 2019-12-21T16:06:09.778 INFO:tasks.workunit.client.0.smithi172.stderr:119575: waiting osd.0 seq 21474836511 2019-12-21T16:06:09.778 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 0 2019-12-21T16:06:09.778 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 21474836512 -lt 21474836511 2019-12-21T16:06:09.778 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: flush_pg_stats: for s in $seqs 2019-12-21T16:06:09.778 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: echo 1-292057776131 2019-12-21T16:06:09.778 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: cut -d - -f 1 2019-12-21T16:06:09.779 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: osd=1 2019-12-21T16:06:09.779 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: echo 1-292057776131 2019-12-21T16:06:09.779 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: cut -d - -f 2 2019-12-21T16:06:09.779 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: seq=292057776131 2019-12-21T16:06:09.779 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1989: flush_pg_stats: echo 'waiting osd.1 seq 292057776131' 2019-12-21T16:06:09.779 INFO:tasks.workunit.client.0.smithi172.stderr:119575: waiting osd.1 seq 292057776131 2019-12-21T16:06:09.780 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 1 2019-12-21T16:06:09.780 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 292057776131 -lt 292057776131 2019-12-21T16:06:09.780 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: flush_pg_stats: for s in $seqs 2019-12-21T16:06:09.780 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: echo 2-292057776131 2019-12-21T16:06:09.780 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: cut -d - -f 1 2019-12-21T16:06:09.780 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: osd=2 2019-12-21T16:06:09.780 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: echo 2-292057776131 2019-12-21T16:06:09.781 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: cut -d - -f 2 2019-12-21T16:06:09.781 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: seq=292057776131 2019-12-21T16:06:09.781 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1989: flush_pg_stats: echo 'waiting osd.2 seq 292057776131' 2019-12-21T16:06:09.781 INFO:tasks.workunit.client.0.smithi172.stderr:119575: waiting osd.2 seq 292057776131 2019-12-21T16:06:09.781 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 2 2019-12-21T16:06:09.781 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 292057776131 -lt 292057776131 2019-12-21T16:06:09.781 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: flush_pg_stats: for s in $seqs 2019-12-21T16:06:09.782 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: echo 3-249108103178 2019-12-21T16:06:09.782 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: cut -d - -f 1 2019-12-21T16:06:09.782 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: osd=3 2019-12-21T16:06:09.782 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: echo 3-249108103178 2019-12-21T16:06:09.782 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: cut -d - -f 2 2019-12-21T16:06:09.782 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: seq=249108103178 2019-12-21T16:06:09.782 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1989: flush_pg_stats: echo 'waiting osd.3 seq 249108103178' 2019-12-21T16:06:09.783 INFO:tasks.workunit.client.0.smithi172.stderr:119575: waiting osd.3 seq 249108103178 2019-12-21T16:06:09.783 INFO:tasks.workunit.client.0.smithi172.stderr:1191474836512 2019-12-21T16:06:09.783 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: cut -d - -f 2 2019-12-21T16:06:09.783 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: seq=21474836512 2019-12-21T16:06:09.783 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1989: flush_pg_stats: echo 'waiting osd.0 seq 21474836512' 2019-12-21T16:06:09.784 INFO:tasks.workunit.client.0.smithi172.stderr:119575: waiting osd.0 seq 21474836512 2019-12-21T16:06:09.784 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 0 2019-12-21T16:06:09.784 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 21474836512 -lt 21474836512 2019-12-21T16:06:09.784 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: flush_pg_stats: for s in $seqs 2019-12-21T16:06:09.784 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: echo 1-292057776130 2019-12-21T16:06:09.785 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: cut -d - -f 1 2019-12-21T16:06:09.785 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: osd=1 2019-12-21T16:06:09.785 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: echo 1-292057776130 2019-12-21T16:06:09.785 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: cut -d - -f 2 2019-12-21T16:06:09.785 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: seq=292057776130 2019-12-21T16:06:09.785 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1989: flush_pg_stats: echo 'waiting osd.1 seq 292057776130' 2019-12-21T16:06:09.786 INFO:tasks.workunit.client.0.smithi172.stderr:119575: waiting osd.1 seq 292057776130 2019-12-21T16:06:09.786 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 1 2019-12-21T16:06:09.786 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 292057776131 -lt 292057776130 2019-12-21T16:06:09.786 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: flush_pg_stats: for s in $seqs 2019-12-21T16:06:09.786 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: echo 2-292057776130 2019-12-21T16:06:09.786 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: cut -d - -f 1 2019-12-21T16:06:09.787 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: osd=2 2019-12-21T16:06:09.787 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: echo 2-292057776130 2019-12-21T16:06:09.787 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: cut -d - -f 2 2019-12-21T16:06:09.787 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: seq=292057776130 2019-12-21T16:06:09.787 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1989: flush_pg_stats: echo 'waiting osd.2 seq 292057776130' 2019-12-21T16:06:09.788 INFO:tasks.workunit.client.0.smithi172.stderr:119575: waiting osd.2 seq 292057776130 2019-12-21T16:06:09.788 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 2 2019-12-21T16:06:09.788 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 292057776131 -lt 292057776130 2019-12-21T16:06:09.788 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: flush_pg_stats: for s in $seqs 2019-12-21T16:06:09.788 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: echo 3-249108103179 2019-12-21T16:06:09.788 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: cut -d - -f 1 2019-12-21T16:06:09.788 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: osd=3 2019-12-21T16:06:09.789 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: echo 3-249108103179 2019-12-21T16:06:09.789 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: cut -d - -f 2 2019-12-21T16:06:09.789 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: seq=249108103179 2019-12-21T16:06:09.789 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1989: flush_pg_stats: echo 'waiting osd.3 seq 249108103179' 2019-12-21T16:06:09.789 INFO:tasks.workunit.client.0.smithi172.stderr:119575: waiting osd.3 seq 249108103179 2019-12-21T16:06:12.059 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 3 2019-12-21T16:06:12.059 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 249108103177 -lt 249108103179 2019-12-21T16:06:12.059 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1991: flush_pg_stats: sleep 1 2019-12-21T16:06:12.060 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1992: flush_pg_stats: '[' 300 -eq 0 ']' 2019-12-21T16:06:12.060 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 3 2019-12-21T16:06:12.060 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 249108103180 -lt 249108103179 2019-12-21T16:06:12.060 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1555: wait_for_clean: get_num_pgs 2019-12-21T16:06:12.060 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: ceph --format json status 2019-12-21T16:06:12.060 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: jq .pgmap.num_pgs 2019-12-21T16:06:12.061 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1555: wait_for_clean: test 5 == 0 2019-12-21T16:06:12.061 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1559: wait_for_clean: true 2019-12-21T16:06:12.061 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1563: wait_for_clean: get_num_active_clean 2019-12-21T16:06:12.061 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: get_num_active_clean: local expression 2019-12-21T16:06:12.061 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2019-12-21T16:06:12.061 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("stale") | not)' 2019-12-21T16:06:12.061 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: ceph --format json pg dump pgs 2019-12-21T16:06:12.061 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2019-12-21T16:06:12.062 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1563: wait_for_clean: cur_active_clean=5 2019-12-21T16:06:12.062 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1564: wait_for_clean: get_num_pgs 2019-12-21T16:06:12.062 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: ceph --format json status 2019-12-21T16:06:12.062 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: jq .pgmap.num_pgs 2019-12-21T16:06:12.062 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1564: wait_for_clean: test 5 = 5 2019-12-21T16:06:12.062 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1564: wait_for_clean: break 2019-12-21T16:06:12.062 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: return 0 2019-12-21T16:06:12.063 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1911: run_in_background: return 0 2019-12-21T16:06:12.071 INFO:tasks.workunit.client.0.smithi172.stderr:575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 3 2019-12-21T16:06:12.071 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 249108103177 -lt 249108103178 2019-12-21T16:06:12.071 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1991: flush_pg_stats: sleep 1 2019-12-21T16:06:12.071 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1992: flush_pg_stats: '[' 300 -eq 0 ']' 2019-12-21T16:06:12.071 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 3 2019-12-21T16:06:12.071 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 249108103180 -lt 249108103178 2019-12-21T16:06:12.072 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1555: wait_for_clean: get_num_pgs 2019-12-21T16:06:12.072 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: ceph --format json status 2019-12-21T16:06:12.072 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: jq .pgmap.num_pgs 2019-12-21T16:06:12.072 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1555: wait_for_clean: test 5 == 0 2019-12-21T16:06:12.072 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1559: wait_for_clean: true 2019-12-21T16:06:12.072 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1563: wait_for_clean: get_num_active_clean 2019-12-21T16:06:12.072 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: get_num_active_clean: local expression 2019-12-21T16:06:12.073 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2019-12-21T16:06:12.073 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("stale") | not)' 2019-12-21T16:06:12.073 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: ceph --format json pg dump pgs 2019-12-21T16:06:12.073 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2019-12-21T16:06:12.073 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1563: wait_for_clean: cur_active_clean=5 2019-12-21T16:06:12.073 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1564: wait_for_clean: get_num_pgs 2019-12-21T16:06:12.073 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: ceph --format json status 2019-12-21T16:06:12.074 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: jq .pgmap.num_pgs 2019-12-21T16:06:12.074 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1564: wait_for_clean: test 5 = 5 2019-12-21T16:06:12.074 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1564: wait_for_clean: break 2019-12-21T16:06:12.074 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: return 0 2019-12-21T16:06:12.074 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1911: run_in_background: return 0 2019-12-21T16:06:12.075 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1946: wait_background: for pid in $pids 2019-12-21T16:06:12.075 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1947: wait_background: wait 160843 2019-12-21T16:06:12.076 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1954: wait_background: eval 'pids='\'''\''' 2019-12-21T16:06:12.076 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1954: wait_background: pids= 2019-12-21T16:06:12.076 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1956: wait_background: return 0 2019-12-21T16:06:12.076 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:218: corrupt_and_repair_two: return_code=0 2019-12-21T16:06:12.076 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:219: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2019-12-21T16:06:12.076 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:221: corrupt_and_repair_two: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2019-12-21T16:06:12.095 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:222: corrupt_and_repair_two: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2019-12-21T16:06:12.099 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:269: corrupt_and_repair_erasure_coded: corrupt_and_repair_two td/osd-scrub-repair ecpool 3 1 2019-12-21T16:06:12.099 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:191: corrupt_and_repair_two: local dir=td/osd-scrub-repair 2019-12-21T16:06:12.099 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:192: corrupt_and_repair_two: local poolname=ecpool 2019-12-21T16:06:12.099 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:193: corrupt_and_repair_two: local first=3 2019-12-21T16:06:12.100 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:194: corrupt_and_repair_two: local second=1 2019-12-21T16:06:12.100 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:199: corrupt_and_repair_two: pids= 2019-12-21T16:06:12.100 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:200: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2019-12-21T16:06:12.100 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1907: run_in_background: local pid_variable=pids 2019-12-21T16:06:12.100 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1908: run_in_background: shift 2019-12-21T16:06:12.100 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1912: run_in_background: eval 'pids+=" 162252"' 2019-12-21T16:06:12.100 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1912: run_in_background: pids+=' 162252' 2019-12-21T16:06:12.101 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:201: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2019-12-21T16:06:12.101 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1907: run_in_background: local pid_variable=pids 2019-12-21T16:06:12.101 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1908: run_in_background: shift 2019-12-21T16:06:12.101 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1912: run_in_background: eval 'pids+=" 162254"' 2019-12-21T16:06:12.101 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1912: run_in_background: pids+=' 162254' 2019-12-21T16:06:12.101 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:202: corrupt_and_repair_two: wait_background pids 2019-12-21T16:06:12.101 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1943: wait_background: pids=' 162252 162254' 2019-12-21T16:06:12.102 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1911: run_in_background: objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2019-12-21T16:06:12.102 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1945: wait_background: return_code=0 2019-12-21T16:06:12.102 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1946: wait_background: for pid in $pids 2019-12-21T16:06:12.102 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1947: wait_background: wait 162252 2019-12-21T16:06:12.102 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1911: run_in_background: sed 's/^/119575: /' 2019-12-21T16:06:12.102 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1911: run_in_background: objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2019-12-21T16:06:12.103 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1911: run_in_background: sed 's/^/119575: /' 2019-12-21T16:06:13.376 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: objectstore_tool: local dir=td/osd-scrub-repair 2019-12-21T16:06:13.377 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: objectstore_tool: shift 2019-12-21T16:06:13.377 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1235: objectstore_tool: local id=1 2019-12-21T16:06:13.377 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1236: objectstore_tool: shift 2019-12-21T16:06:13.377 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1238: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING remove 2019-12-21T16:06:13.377 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1205: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2019-12-21T16:06:13.377 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nowait: shift 2019-12-21T16:06:13.377 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1207: _objectstore_tool_nowait: local id=1 2019-12-21T16:06:13.378 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1208: _objectstore_tool_nowait: shift 2019-12-21T16:06:13.378 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1210: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2019-12-21T16:06:13.378 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2019-12-21T16:06:13.378 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2019-12-21T16:06:13.378 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2019-12-21T16:06:13.378 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2019-12-21T16:06:13.378 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2019-12-21T16:06:13.379 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2019-12-21T16:06:13.379 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1212: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING remove 2019-12-21T16:06:13.379 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1193: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2019-12-21T16:06:13.379 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1194: _objectstore_tool_nodown: shift 2019-12-21T16:06:13.379 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1195: _objectstore_tool_nodown: local id=1 2019-12-21T16:06:13.379 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: _objectstore_tool_nodown: shift 2019-12-21T16:06:13.379 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2019-12-21T16:06:13.380 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING remove 2019-12-21T16:06:13.380 INFO:tasks.workunit.client.0.smithi172.stderr:119575: remove 1#2:eb822e21:::SOMETHING:head# 2019-12-21T16:06:13.380 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1213: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2019-12-21T16:06:13.380 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:842: activate_osd: local dir=td/osd-scrub-repair 2019-12-21T16:06:13.380 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: shift 2019-12-21T16:06:13.380 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: local id=1 2019-12-21T16:06:13.380 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: shift 2019-12-21T16:06:13.381 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local osd_data=td/osd-scrub-repair/1 2019-12-21T16:06:13.381 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local 'ceph_args=--fsid=973e3fc1-e63e-4bde-9db6-84c68fd5e93e --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2019-12-21T16:06:13.381 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2019-12-21T16:06:13.381 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: 119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: objectstore_tool: local dir=td/osd-scrub-repair 2019-12-21T16:06:13.381 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: objectstore_tool: shift 2019-12-21T16:06:13.381 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1235: objectstore_tool: local id=3 2019-12-21T16:06:13.381 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1236: objectstore_tool: shift 2019-12-21T16:06:13.381 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1238: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING remove 2019-12-21T16:06:13.382 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1205: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2019-12-21T16:06:13.382 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nowait: shift 2019-12-21T16:06:13.382 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1207: _objectstore_tool_nowait: local id=3 2019-12-21T16:06:13.382 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1208: _objectstore_tool_nowait: shift 2019-12-21T16:06:13.382 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1210: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2019-12-21T16:06:13.382 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2019-12-21T16:06:13.382 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2019-12-21T16:06:13.383 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2019-12-21T16:06:13.383 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2019-12-21T16:06:13.383 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2019-12-21T16:06:13.383 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2019-12-21T16:06:13.383 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1212: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING remove 2019-12-21T16:06:13.383 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1193: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2019-12-21T16:06:13.383 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1194: _objectstore_tool_nodown: shift 2019-12-21T16:06:13.384 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1195: _objectstore_tool_nodown: local id=3 2019-12-21T16:06:13.384 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: _objectstore_tool_nodown: shift 2019-12-21T16:06:13.384 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2019-12-21T16:06:13.384 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING remove 2019-12-21T16:06:13.384 INFO:tasks.workunit.client.0.smithi172.stderr:119575: remove 0#2:eb822e21:::SOMETHING:head# 2019-12-21T16:06:13.384 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1213: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2019-12-21T16:06:13.384 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:842: activate_osd: local dir=td/osd-scrub-repair 2019-12-21T16:06:13.385 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: shift 2019-12-21T16:06:13.385 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: local id=3 2019-12-21T16:06:13.385 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: shift 2019-12-21T16:06:13.385 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local osd_data=td/osd-scrub-repair/3 2019-12-21T16:06:13.385 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local 'ceph_args=--fsid=973e3fc1-e63e-4bde-9db6-84c68fd5e93e --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2019-12-21T16:06:13.385 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2019-12-21T16:06:13.386 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-journal-size=100' 2019-12-21T16:06:13.386 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2019-12-21T16:06:13.386 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2019-12-21T16:06:13.386 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2019-12-21T16:06:13.386 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --chdir=' 2019-12-21T16:06:13.387 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+= 2019-12-21T16:06:13.387 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2019-12-21T16:06:13.387 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: get_asok_path 2019-12-21T16:06:13.387 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2019-12-21T16:06:13.387 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2019-12-21T16:06:13.387 INFO:tasks.workunit.client.0.smithi172.stderr:119575: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2019-12-21T16:06:13.387 INFO:tasks.workunit.client.0.smithi172.stderr:119575: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T16:06:13.388 INFO:tasks.workunit.client.0.smithi172.stderr:119575: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.119575 2019-12-21T16:06:13.388 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.119575/$cluster-$name.asok' 2019-12-21T16:06:13.388 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.119575/$cluster-$name.asok' 2019-12-21T16:06:13.388 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --debug-osd=20' 2019-12-21T16:06:13.388 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2019-12-21T16:06:13.388 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2019-12-21T16:06:13.389 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2019-12-21T16:06:13.389 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2019-12-21T16:06:13.389 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2019-12-21T16:06:13.389 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' ' 2019-12-21T16:06:13.389 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+= 2019-12-21T16:06:13.389 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: mkdir -p td/osd-scrub-repair/1 2019-12-21T16:06:13.389 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: echo start osd.1 2019-12-21T16:06:13.390 INFO:tasks.workunit.client.0.smithi172.stderr:119575: start osd.1 2019-12-21T16:06:13.390 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph-osd -i 1 --fsid=973e3fc1-e63e-4bde-9db6-84c68fd5e93e --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.119575/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2019-12-21T16:06:13.390 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: cat td/osd-scrub-repair/1/whoami 2019-12-21T16:06:13.390 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standaactivate_osd: ceph_args+=' --osd-journal-size=100' 2019-12-21T16:06:13.390 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2019-12-21T16:06:13.391 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2019-12-21T16:06:13.391 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2019-12-21T16:06:13.391 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --chdir=' 2019-12-21T16:06:13.391 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+= 2019-12-21T16:06:13.392 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2019-12-21T16:06:13.392 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: get_asok_path 2019-12-21T16:06:13.392 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2019-12-21T16:06:13.392 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2019-12-21T16:06:13.392 INFO:tasks.workunit.client.0.smithi172.stderr:119575: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2019-12-21T16:06:13.392 INFO:tasks.workunit.client.0.smithi172.stderr:119575: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T16:06:13.392 INFO:tasks.workunit.client.0.smithi172.stderr:119575: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.119575 2019-12-21T16:06:13.393 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.119575/$cluster-$name.asok' 2019-12-21T16:06:13.393 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.119575/$cluster-$name.asok' 2019-12-21T16:06:13.393 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --debug-osd=20' 2019-12-21T16:06:13.393 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2019-12-21T16:06:13.393 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2019-12-21T16:06:13.393 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2019-12-21T16:06:13.393 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2019-12-21T16:06:13.394 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2019-12-21T16:06:13.394 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' ' 2019-12-21T16:06:13.394 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+= 2019-12-21T16:06:13.394 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: mkdir -p td/osd-scrub-repair/3 2019-12-21T16:06:13.394 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: echo start osd.3 2019-12-21T16:06:13.395 INFO:tasks.workunit.client.0.smithi172.stderr:119575: start osd.3 2019-12-21T16:06:13.395 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph-osd -i 3 --fsid=973e3fc1-e63e-4bde-9db6-84c68fd5e93e --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.119575/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2019-12-21T16:06:13.395 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: cat td/osd-scrub-repair/3/whoami 2019-12-21T16:06:15.111 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: '[' 1 = 1 ']' 2019-12-21T16:06:15.111 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph osd dump --format=json 2019-12-21T16:06:15.111 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: grep -q '"noup"' 2019-12-21T16:06:15.112 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: jq '.flags_set[]' 2019-12-21T16:06:15.112 INFO:tasks.workunit.client.0.smithi172.stderr:119575: 2019-12-21T16:06:13.399+0000 7f91a7110ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T16:06:15.112 INFO:tasks.workunit.client.0.smithi172.stderr:119575: 2019-12-21T16:06:13.402+0000 7f91a7110ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T16:06:15.112 INFO:tasks.workunit.client.0.smithi172.stderr:119575: 2019-12-21T16:06:13.405+0000 7f91a7110ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T16:06:15.112 INFO:tasks.workunit.client.0.smithi172.stderr:119575: 2019-12-21T16:06:13.406+0000 7f91a7110ec0 -1 Falling back to public interface 2019-12-21T16:06:15.112 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: wait_for_osd up 1 2019-12-21T16:06:15.112 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:914: wait_for_osd: local state=up 2019-12-21T16:06:15.113 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local id=1 2019-12-21T16:06:15.113 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: status=1 2019-12-21T16:06:15.113 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i=0 )) 2019-12-21T16:06:15.113 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i < 300 )) 2019-12-21T16:06:15.113 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: echo 0 2019-12-21T16:06:15.113 INFO:tasks.workunit.client.0.smithi172.stderr:119575: 0 2019-12-21T16:06:15.114 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: ceph osd dump 2019-12-21T16:06:15.114 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: grep 'osd.1 up' 2019-12-21T16:06:15.114 INFO:tasks.workunit.client.0.smithi172.stderr:119575: 2019-12-21T16:06:13.831+0000 7f91a7110ec0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2019-12-21T16:06:15.114 INFO:tasks.workunit.client.0.smithi172.stderr:119575: 2019-12-21T16:06:13.842+0000 7f91a7110ec0 -1 journal do_read_entry(1093632): bad header magic 2019-12-21T16:06:15.114 INFO:tasks.workunit.client.0.smithi172.stderr:119575: 2019-12-21T16:06:13.842+0000 7f91a7110ec0 -1 journal do_read_entry(1093632): bad header magic 2019-12-21T16:06:15.114 INFO:tasks.workunit.client.0.smithi172.stderr:119575: 2019-12-21T16:06:13.848+0000 7f91a7110ec0 -1 osd.1 69 log_to_monitors {default=true} 2019-12-21T16:06:15.114 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: sleep 1 2019-12-21T16:06:15.115 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i++ )) 2019-12-21T16:06:15.115 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i < 300 )) 2019-12-21T16:06:15.115 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: echo 1 2019-12-21T16:06:15.115 INFO:tasks.workunit.client.0.smithi172.stderr:119575: 1 2019-12-21T16:06:15.115 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: ceph osd dump 2019-12-21T16:06:15.115 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: grep 'osd.1 up' 2019-12-21T16:06:15.116 INFO:tasks.workunit.client.0.smithi172.stderr:119575: osd.1 up in weight 1 up_from 72 up_thru 72 down_at 70 last_clean_interval [68,69) [v2:127.0.0.1:6822/162402,v1:127.0.0.1:6824/162402] [v2:127.0.0.1:6826/162402,v1:127.0.0.1:6828/162402] exists,up 627d9f35-2c70-4ea0-8821-1ca4dc71496b 2019-12-21T16:06:15.116 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=0 2019-12-21T16:06:15.116 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: break 2019-12-21T16:06:15.116 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: return 0 2019-12-21T16:06:15.116 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1239: objectstore_tool: wait_for_clean 2019-12-21T16:06:15.116 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1548: wait_for_clean: local cmd= 2019-12-21T16:06:15.117 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1549: wait_for_clean: local num_active_clean=-1 2019-12-21T16:06:15.117 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1550: wait_for_clean: local cur_active_clean 2019-12-21T16:06:15.119 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1551: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMElone/ceph-helpers.sh:871: activate_osd: '[' 3 = 3 ']' 2019-12-21T16:06:15.119 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph osd dump --format=json 2019-12-21T16:06:15.119 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: grep -q '"noup"' 2019-12-21T16:06:15.120 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: jq '.flags_set[]' 2019-12-21T16:06:15.120 INFO:tasks.workunit.client.0.smithi172.stderr:119575: 2019-12-21T16:06:13.399+0000 7f4ca5c4cec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T16:06:15.120 INFO:tasks.workunit.client.0.smithi172.stderr:119575: 2019-12-21T16:06:13.402+0000 7f4ca5c4cec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T16:06:15.120 INFO:tasks.workunit.client.0.smithi172.stderr:119575: 2019-12-21T16:06:13.403+0000 7f4ca5c4cec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T16:06:15.120 INFO:tasks.workunit.client.0.smithi172.stderr:119575: 2019-12-21T16:06:13.405+0000 7f4ca5c4cec0 -1 Falling back to public interface 2019-12-21T16:06:15.120 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: wait_for_osd up 3 2019-12-21T16:06:15.121 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:914: wait_for_osd: local state=up 2019-12-21T16:06:15.121 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local id=3 2019-12-21T16:06:15.121 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: status=1 2019-12-21T16:06:15.121 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i=0 )) 2019-12-21T16:06:15.121 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i < 300 )) 2019-12-21T16:06:15.121 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: echo 0 2019-12-21T16:06:15.121 INFO:tasks.workunit.client.0.smithi172.stderr:119575: 0 2019-12-21T16:06:15.122 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: ceph osd dump 2019-12-21T16:06:15.122 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: grep 'osd.3 up' 2019-12-21T16:06:15.122 INFO:tasks.workunit.client.0.smithi172.stderr:119575: 2019-12-21T16:06:13.850+0000 7f4ca5c4cec0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2019-12-21T16:06:15.122 INFO:tasks.workunit.client.0.smithi172.stderr:119575: 2019-12-21T16:06:13.855+0000 7f4ca5c4cec0 -1 journal do_read_entry(1138688): bad header magic 2019-12-21T16:06:15.122 INFO:tasks.workunit.client.0.smithi172.stderr:119575: 2019-12-21T16:06:13.855+0000 7f4ca5c4cec0 -1 journal do_read_entry(1138688): bad header magic 2019-12-21T16:06:15.122 INFO:tasks.workunit.client.0.smithi172.stderr:119575: 2019-12-21T16:06:13.861+0000 7f4ca5c4cec0 -1 osd.3 69 log_to_monitors {default=true} 2019-12-21T16:06:15.122 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: sleep 1 2019-12-21T16:06:15.123 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i++ )) 2019-12-21T16:06:15.123 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i < 300 )) 2019-12-21T16:06:15.123 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: echo 1 2019-12-21T16:06:15.123 INFO:tasks.workunit.client.0.smithi172.stderr:119575: 1 2019-12-21T16:06:15.123 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: ceph osd dump 2019-12-21T16:06:15.123 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: grep 'osd.3 up' 2019-12-21T16:06:15.123 INFO:tasks.workunit.client.0.smithi172.stderr:119575: osd.3 up in weight 1 up_from 72 up_thru 72 down_at 70 last_clean_interval [58,69) [v2:127.0.0.1:6815/162404,v1:127.0.0.1:6817/162404] [v2:127.0.0.1:6819/162404,v1:127.0.0.1:6821/162404] exists,up d2b8f48d-5699-449e-8fc7-707402a64a19 2019-12-21T16:06:15.124 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=0 2019-12-21T16:06:15.124 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: break 2019-12-21T16:06:15.124 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: return 0 2019-12-21T16:06:15.124 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1239: objectstore_tool: wait_for_clean 2019-12-21T16:06:15.124 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1548: wait_for_clean: local cmd= 2019-12-21T16:06:15.124 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1549: wait_for_clean: local num_active_clean=-1 2019-12-21T16:06:15.125 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1550: wait_for_clean: local cur_active_clean 2019-12-21T16:06:15.902 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1551: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2019-12-21T16:06:15.903 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1551: wait_for_clean: get_timeout_delays 90 .1 2019-12-21T16:06:15.903 INFO:tasks.workunit.client.0.smithi172.stderr:119575: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1494: get_timeout_delays: shopt -q -o xtrace 2019-12-21T16:06:15.903 INFO:tasks.workunit.client.0.smithi172.stderr:119575: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1494: get_timeout_delays: echo true 2019-12-21T16:06:15.903 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1494: get_timeout_delays: local trace=true 2019-12-21T16:06:15.903 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1495: get_timeout_delays: true 2019-12-21T16:06:15.904 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1495: get_timeout_delays: shopt -u -o xtrace 2019-12-21T16:06:15.904 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1551: wait_for_clean: local -a delays 2019-12-21T16:06:15.904 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1552: wait_for_clean: local -i loop=0 2019-12-21T16:06:15.904 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1554: wait_for_clean: flush_pg_stats 2019-12-21T16:06:15.904 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1977: flush_pg_stats: local timeout=300 2019-12-21T16:06:15.904 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1979: flush_pg_stats: ceph osd ls 2019-12-21T16:06:15.905 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1979: flush_pg_stats: ids='0 2019-12-21T16:06:15.905 INFO:tasks.workunit.client.0.smithi172.stderr:119575: 1 2019-12-21T16:06:15.905 INFO:tasks.workunit.client.0.smithi172.stderr:119575: 2 2019-12-21T16:06:15.905 INFO:tasks.workunit.client.0.smithi172.stderr:119575: 3' 2019-12-21T16:06:15.905 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1980: flush_pg_stats: seqs= 2019-12-21T16:06:15.905 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: flush_pg_stats: for osd in $ids 2019-12-21T16:06:15.905 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2019-12-21T16:06:15.906 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: seq=21474836515 2019-12-21T16:06:15.906 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: flush_pg_stats: seqs=' 0-21474836515' 2019-12-21T16:06:15.906 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: flush_pg_stats: for osd in $ids 2019-12-21T16:06:15.906 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2019-12-21T16:06:15.906 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: seq=309237645315 2019-12-21T16:06:15.906 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: flush_pg_stats: seqs=' 0-21474836515 1-309237645315' 2019-12-21T16:06:15.907 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: flush_pg_stats: for osd in $ids 2019-12-21T16:06:15.907 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2019-12-21T16:06:15.907 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: seq=292057776134 2019-12-21T16:06:15.907 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: flush_pg_stats: seqs=' 0-21474836515 1-309237645315 2-292057776134' 2019-12-21T16:06:15.907 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: flush_pg_stats: for osd in $ids 2019-12-21T16:06:15.907 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2019-12-21T16:06:15.908 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: seq=309237645314 2019-12-21T16:06:15.908 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: flush_pg_stats: seqs=' 0-21474836515 1-309237645315 2-292057776134 3-309237645314' 2019-12-21T16:06:15.908 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: flush_pg_stats: for s in $seqs 2019-12-21T16:06:15.908 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: echo 0-21474836515 2019-12-21T16:06:15.908 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: cut -d - -f 1 2019-12-21T16:06:15.908 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: osd=0 2019-12-21T16:06:15.909 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: echo 0-214748365OUT .1)) 2019-12-21T16:06:15.909 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1551: wait_for_clean: get_timeout_delays 90 .1 2019-12-21T16:06:15.909 INFO:tasks.workunit.client.0.smithi172.stderr:119575: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1494: get_timeout_delays: shopt -q -o xtrace 2019-12-21T16:06:15.909 INFO:tasks.workunit.client.0.smithi172.stderr:119575: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1494: get_timeout_delays: echo true 2019-12-21T16:06:15.909 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1494: get_timeout_delays: local trace=true 2019-12-21T16:06:15.909 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1495: get_timeout_delays: true 2019-12-21T16:06:15.910 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1495: get_timeout_delays: shopt -u -o xtrace 2019-12-21T16:06:15.910 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1551: wait_for_clean: local -a delays 2019-12-21T16:06:15.910 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1552: wait_for_clean: local -i loop=0 2019-12-21T16:06:15.910 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1554: wait_for_clean: flush_pg_stats 2019-12-21T16:06:15.910 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1977: flush_pg_stats: local timeout=300 2019-12-21T16:06:15.910 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1979: flush_pg_stats: ceph osd ls 2019-12-21T16:06:15.911 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1979: flush_pg_stats: ids='0 2019-12-21T16:06:15.911 INFO:tasks.workunit.client.0.smithi172.stderr:119575: 1 2019-12-21T16:06:15.911 INFO:tasks.workunit.client.0.smithi172.stderr:119575: 2 2019-12-21T16:06:15.911 INFO:tasks.workunit.client.0.smithi172.stderr:119575: 3' 2019-12-21T16:06:15.911 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1980: flush_pg_stats: seqs= 2019-12-21T16:06:15.911 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: flush_pg_stats: for osd in $ids 2019-12-21T16:06:15.912 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2019-12-21T16:06:15.912 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: seq=21474836514 2019-12-21T16:06:15.912 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: flush_pg_stats: seqs=' 0-21474836514' 2019-12-21T16:06:15.912 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: flush_pg_stats: for osd in $ids 2019-12-21T16:06:15.912 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2019-12-21T16:06:15.912 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: seq=309237645314 2019-12-21T16:06:15.913 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: flush_pg_stats: seqs=' 0-21474836514 1-309237645314' 2019-12-21T16:06:15.913 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: flush_pg_stats: for osd in $ids 2019-12-21T16:06:15.913 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2019-12-21T16:06:15.913 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: seq=292057776133 2019-12-21T16:06:15.913 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: flush_pg_stats: seqs=' 0-21474836514 1-309237645314 2-292057776133' 2019-12-21T16:06:15.913 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: flush_pg_stats: for osd in $ids 2019-12-21T16:06:15.913 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2019-12-21T16:06:15.914 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: seq=309237645315 2019-12-21T16:06:15.914 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: flush_pg_stats: seqs=' 0-21474836514 1-309237645314 2-292057776133 3-309237645315' 2019-12-21T16:06:15.914 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: flush_pg_stats: for s in $seqs 2019-12-21T16:06:15.914 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: echo 0-21474836514 2019-12-21T16:06:15.914 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: cut -d - -f 1 2019-12-21T16:06:15.914 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: osd=0 2019-12-21T16:06:17.918 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: echo 0-21474836515 2019-12-21T16:06:17.919 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: cut -d - -f 2 2019-12-21T16:06:17.919 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: seq=21474836515 2019-12-21T16:06:17.919 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1989: flush_pg_stats: echo 'waiting osd.0 seq 21474836515' 2019-12-21T16:06:17.919 INFO:tasks.workunit.client.0.smithi172.stderr:119575: waiting osd.0 seq 21474836515 2019-12-21T16:06:17.919 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 0 2019-12-21T16:06:17.919 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 21474836513 -lt 21474836515 2019-12-21T16:06:17.920 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1991: flush_pg_stats: sleep 1 2019-12-21T16:06:17.920 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1992: flush_pg_stats: '[' 300 -eq 0 ']' 2019-12-21T16:06:17.920 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 0 2019-12-21T16:06:17.920 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 21474836515 -lt 21474836515 2019-12-21T16:06:17.920 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: flush_pg_stats: for s in $seqs 2019-12-21T16:06:17.920 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: echo 1-309237645315 2019-12-21T16:06:17.921 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: cut -d - -f 1 2019-12-21T16:06:17.921 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: osd=1 2019-12-21T16:06:17.921 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: echo 1-309237645315 2019-12-21T16:06:17.921 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: cut -d - -f 2 2019-12-21T16:06:17.921 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: seq=309237645315 2019-12-21T16:06:17.921 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1989: flush_pg_stats: echo 'waiting osd.1 seq 309237645315' 2019-12-21T16:06:17.921 INFO:tasks.workunit.client.0.smithi172.stderr:119575: waiting osd.1 seq 309237645315 2019-12-21T16:06:17.922 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 1 2019-12-21T16:06:17.922 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 309237645315 -lt 309237645315 2019-12-21T16:06:17.922 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: flush_pg_stats: for s in $seqs 2019-12-21T16:06:17.922 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: echo 2-292057776134 2019-12-21T16:06:17.922 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: cut -d - -f 1 2019-12-21T16:06:17.922 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: osd=2 2019-12-21T16:06:17.923 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: echo 2-292057776134 2019-12-21T16:06:17.923 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: cut -d - -f 2 2019-12-21T16:06:17.923 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: seq=292057776134 2019-12-21T16:06:17.923 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1989: flush_pg_stats: echo 'waiting osd.2 seq 292057776134' 2019-12-21T16:06:17.923 INFO:tasks.workunit.client.0.smithi172.stderr:119575: waiting osd.2 seq 292057776134 2019-12-21T16:06:17.923 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 2 2019-12-21T16:06:17.923 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 292057776135 -lt 292057776134 2019-12-21T16:06:17.924 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: flush_pg_stats: for s in $seqs 2019-12-21T16:06:17.924 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: echo 3-309237645314 2019-12-21T16:06:17.924 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: cut -d - -f 1 2019-12-21T16:06:17.924 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: osd=3 2019-12-21T16:06:17.925 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standal14 2019-12-21T16:06:17.925 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: cut -d - -f 2 2019-12-21T16:06:17.925 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: seq=21474836514 2019-12-21T16:06:17.925 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1989: flush_pg_stats: echo 'waiting osd.0 seq 21474836514' 2019-12-21T16:06:17.925 INFO:tasks.workunit.client.0.smithi172.stderr:119575: waiting osd.0 seq 21474836514 2019-12-21T16:06:17.926 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 0 2019-12-21T16:06:17.926 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 21474836513 -lt 21474836514 2019-12-21T16:06:17.926 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1991: flush_pg_stats: sleep 1 2019-12-21T16:06:17.926 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1992: flush_pg_stats: '[' 300 -eq 0 ']' 2019-12-21T16:06:17.926 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 0 2019-12-21T16:06:17.926 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 21474836515 -lt 21474836514 2019-12-21T16:06:17.926 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: flush_pg_stats: for s in $seqs 2019-12-21T16:06:17.926 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: echo 1-309237645314 2019-12-21T16:06:17.927 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: cut -d - -f 1 2019-12-21T16:06:17.927 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: osd=1 2019-12-21T16:06:17.927 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: echo 1-309237645314 2019-12-21T16:06:17.927 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: cut -d - -f 2 2019-12-21T16:06:17.927 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: seq=309237645314 2019-12-21T16:06:17.927 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1989: flush_pg_stats: echo 'waiting osd.1 seq 309237645314' 2019-12-21T16:06:17.927 INFO:tasks.workunit.client.0.smithi172.stderr:119575: waiting osd.1 seq 309237645314 2019-12-21T16:06:17.928 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 1 2019-12-21T16:06:17.928 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 309237645315 -lt 309237645314 2019-12-21T16:06:17.928 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: flush_pg_stats: for s in $seqs 2019-12-21T16:06:17.928 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: echo 2-292057776133 2019-12-21T16:06:17.928 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: cut -d - -f 1 2019-12-21T16:06:17.928 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: osd=2 2019-12-21T16:06:17.928 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: echo 2-292057776133 2019-12-21T16:06:17.929 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: cut -d - -f 2 2019-12-21T16:06:17.929 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: seq=292057776133 2019-12-21T16:06:17.929 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1989: flush_pg_stats: echo 'waiting osd.2 seq 292057776133' 2019-12-21T16:06:17.929 INFO:tasks.workunit.client.0.smithi172.stderr:119575: waiting osd.2 seq 292057776133 2019-12-21T16:06:17.929 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 2 2019-12-21T16:06:17.929 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 292057776135 -lt 292057776133 2019-12-21T16:06:17.929 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: flush_pg_stats: for s in $seqs 2019-12-21T16:06:17.930 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: echo 3-309237645315 2019-12-21T16:06:17.930 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: cut -d - -f 1 2019-12-21T16:06:17.930 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: osd=3 2019-12-21T16:06:18.956 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: echo 3-309237645314 2019-12-21T16:06:18.957 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: cut -d - -f 2 2019-12-21T16:06:18.957 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: seq=309237645314 2019-12-21T16:06:18.957 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1989: flush_pg_stats: echo 'waiting osd.3 seq 309237645314' 2019-12-21T16:06:18.957 INFO:tasks.workunit.client.0.smithi172.stderr:119575: waiting osd.3 seq 309237645314 2019-12-21T16:06:18.957 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 3 2019-12-21T16:06:18.958 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 309237645315 -lt 309237645314 2019-12-21T16:06:18.958 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1555: wait_for_clean: get_num_pgs 2019-12-21T16:06:18.958 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: ceph --format json status 2019-12-21T16:06:18.958 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: jq .pgmap.num_pgs 2019-12-21T16:06:18.958 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1555: wait_for_clean: test 5 == 0 2019-12-21T16:06:18.958 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1559: wait_for_clean: true 2019-12-21T16:06:18.959 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1563: wait_for_clean: get_num_active_clean 2019-12-21T16:06:18.959 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: get_num_active_clean: local expression 2019-12-21T16:06:18.959 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2019-12-21T16:06:18.959 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("stale") | not)' 2019-12-21T16:06:18.959 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: ceph --format json pg dump pgs 2019-12-21T16:06:18.959 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2019-12-21T16:06:18.959 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1563: wait_for_clean: cur_active_clean=5 2019-12-21T16:06:18.960 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1564: wait_for_clean: get_num_pgs 2019-12-21T16:06:18.960 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: ceph --format json status 2019-12-21T16:06:18.960 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: jq .pgmap.num_pgs 2019-12-21T16:06:18.960 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1564: wait_for_clean: test 5 = 5 2019-12-21T16:06:18.960 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1564: wait_for_clean: break 2019-12-21T16:06:18.960 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: return 0 2019-12-21T16:06:18.961 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1911: run_in_background: return 0 2019-12-21T16:06:18.961 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1946: wait_background: for pid in $pids 2019-12-21T16:06:18.961 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1947: wait_background: wait 162254 2019-12-21T16:06:18.962 INFO:tasks.workunit.client.0.smithi172.stderr:one/ceph-helpers.sh:1988: flush_pg_stats: echo 3-309237645315 2019-12-21T16:06:18.963 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: cut -d - -f 2 2019-12-21T16:06:18.963 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: seq=309237645315 2019-12-21T16:06:18.963 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1989: flush_pg_stats: echo 'waiting osd.3 seq 309237645315' 2019-12-21T16:06:18.963 INFO:tasks.workunit.client.0.smithi172.stderr:119575: waiting osd.3 seq 309237645315 2019-12-21T16:06:18.964 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 3 2019-12-21T16:06:18.964 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 309237645315 -lt 309237645315 2019-12-21T16:06:18.964 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1555: wait_for_clean: get_num_pgs 2019-12-21T16:06:18.964 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: ceph --format json status 2019-12-21T16:06:18.964 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: jq .pgmap.num_pgs 2019-12-21T16:06:18.964 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1555: wait_for_clean: test 5 == 0 2019-12-21T16:06:18.965 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1559: wait_for_clean: true 2019-12-21T16:06:18.965 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1563: wait_for_clean: get_num_active_clean 2019-12-21T16:06:18.965 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: get_num_active_clean: local expression 2019-12-21T16:06:18.965 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2019-12-21T16:06:18.965 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("stale") | not)' 2019-12-21T16:06:18.965 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: ceph --format json pg dump pgs 2019-12-21T16:06:18.966 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2019-12-21T16:06:18.966 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1563: wait_for_clean: cur_active_clean=5 2019-12-21T16:06:18.966 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1564: wait_for_clean: get_num_pgs 2019-12-21T16:06:18.966 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: ceph --format json status 2019-12-21T16:06:18.966 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: jq .pgmap.num_pgs 2019-12-21T16:06:18.966 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1564: wait_for_clean: test 5 = 5 2019-12-21T16:06:18.967 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1564: wait_for_clean: break 2019-12-21T16:06:18.967 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: return 0 2019-12-21T16:06:18.967 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1911: run_in_background: return 0 2019-12-21T16:06:18.967 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1954: wait_background: eval 'pids='\'''\''' 2019-12-21T16:06:18.967 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1954: wait_background: pids= 2019-12-21T16:06:18.967 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1956: wait_background: return 0 2019-12-21T16:06:18.968 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: return_code=0 2019-12-21T16:06:18.968 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:204: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2019-12-21T16:06:18.968 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:209: corrupt_and_repair_two: get_pg ecpool SOMETHING 2019-12-21T16:06:18.968 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1029: get_pg: local poolname=ecpool 2019-12-21T16:06:18.968 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1030: get_pg: local objectname=SOMETHING 2019-12-21T16:06:18.968 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1032: get_pg: ceph --format json osd map ecpool SOMETHING 2019-12-21T16:06:18.969 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1032: get_pg: jq -r .pgid 2019-12-21T16:06:19.215 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:209: corrupt_and_repair_two: local pg=2.0 2019-12-21T16:06:19.215 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:210: corrupt_and_repair_two: repair 2.0 2019-12-21T16:06:19.215 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: repair: local pgid=2.0 2019-12-21T16:06:19.216 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1665: repair: get_last_scrub_stamp 2.0 2019-12-21T16:06:19.216 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=2.0 2019-12-21T16:06:19.216 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T16:06:19.216 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T16:06:19.216 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2019-12-21T16:06:19.456 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1665: repair: local last_scrub=2019-12-21T16:06:04.405865+0000 2019-12-21T16:06:19.456 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1666: repair: ceph pg repair 2.0 2019-12-21T16:06:19.685 INFO:tasks.workunit.client.0.smithi172.stderr:instructing pg 2.0s0 on osd.3 to repair 2019-12-21T16:06:19.693 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: repair: wait_for_scrub 2.0 2019-12-21T16:06:04.405865+0000 2019-12-21T16:06:19.694 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1789: wait_for_scrub: local pgid=2.0 2019-12-21T16:06:19.694 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1790: wait_for_scrub: local last_scrub=2019-12-21T16:06:04.405865+0000 2019-12-21T16:06:19.694 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1791: wait_for_scrub: local sname=last_scrub_stamp 2019-12-21T16:06:19.694 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i=0 )) 2019-12-21T16:06:19.694 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T16:06:19.694 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2019-12-21T16:06:19.694 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=2.0 2019-12-21T16:06:19.695 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T16:06:19.695 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T16:06:19.695 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2019-12-21T16:06:19.917 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T16:06:04.405865+0000 '>' 2019-12-21T16:06:04.405865+0000 2019-12-21T16:06:19.917 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T16:06:20.918 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T16:06:20.918 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T16:06:20.919 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2019-12-21T16:06:20.919 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=2.0 2019-12-21T16:06:20.919 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T16:06:20.919 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T16:06:20.920 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2019-12-21T16:06:21.152 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T16:06:04.405865+0000 '>' 2019-12-21T16:06:04.405865+0000 2019-12-21T16:06:21.152 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T16:06:22.153 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T16:06:22.154 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T16:06:22.154 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2019-12-21T16:06:22.154 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=2.0 2019-12-21T16:06:22.154 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T16:06:22.154 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T16:06:22.154 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2019-12-21T16:06:22.387 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T16:06:04.405865+0000 '>' 2019-12-21T16:06:04.405865+0000 2019-12-21T16:06:22.387 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T16:06:23.388 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T16:06:23.388 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T16:06:23.389 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2019-12-21T16:06:23.389 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=2.0 2019-12-21T16:06:23.389 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T16:06:23.389 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T16:06:23.389 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2019-12-21T16:06:23.618 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T16:06:04.405865+0000 '>' 2019-12-21T16:06:04.405865+0000 2019-12-21T16:06:23.618 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T16:06:24.619 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T16:06:24.619 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T16:06:24.620 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2019-12-21T16:06:24.620 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=2.0 2019-12-21T16:06:24.620 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T16:06:24.620 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T16:06:24.620 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2019-12-21T16:06:24.853 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T16:06:04.405865+0000 '>' 2019-12-21T16:06:04.405865+0000 2019-12-21T16:06:24.853 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T16:06:25.854 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T16:06:25.854 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T16:06:25.855 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2019-12-21T16:06:25.855 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=2.0 2019-12-21T16:06:25.855 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T16:06:25.855 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T16:06:25.855 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2019-12-21T16:06:26.090 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T16:06:19.881623+0000 '>' 2019-12-21T16:06:04.405865+0000 2019-12-21T16:06:26.090 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1795: wait_for_scrub: return 0 2019-12-21T16:06:26.090 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:214: corrupt_and_repair_two: pids= 2019-12-21T16:06:26.090 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:215: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2019-12-21T16:06:26.090 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1907: run_in_background: local pid_variable=pids 2019-12-21T16:06:26.090 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1908: run_in_background: shift 2019-12-21T16:06:26.091 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1912: run_in_background: eval 'pids+=" 163886"' 2019-12-21T16:06:26.091 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1912: run_in_background: pids+=' 163886' 2019-12-21T16:06:26.091 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:216: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2019-12-21T16:06:26.091 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1907: run_in_background: local pid_variable=pids 2019-12-21T16:06:26.091 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1908: run_in_background: shift 2019-12-21T16:06:26.091 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1911: run_in_background: objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2019-12-21T16:06:26.091 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1912: run_in_background: eval 'pids+=" 163888"' 2019-12-21T16:06:26.092 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1912: run_in_background: pids+=' 163888' 2019-12-21T16:06:26.092 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:217: corrupt_and_repair_two: wait_background pids 2019-12-21T16:06:26.092 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1911: run_in_background: sed 's/^/119575: /' 2019-12-21T16:06:26.092 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1943: wait_background: pids=' 163886 163888' 2019-12-21T16:06:26.092 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1945: wait_background: return_code=0 2019-12-21T16:06:26.092 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1946: wait_background: for pid in $pids 2019-12-21T16:06:26.092 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1947: wait_background: wait 163886 2019-12-21T16:06:26.093 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1911: run_in_background: objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2019-12-21T16:06:26.093 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1911: run_in_background: sed 's/^/119575: /' 2019-12-21T16:06:26.768 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: objectstore_tool: local dir=td/osd-scrub-repair 2019-12-21T16:06:26.768 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: objectstore_tool: shift 2019-12-21T16:06:26.769 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1235: objectstore_tool: local id=1 2019-12-21T16:06:26.769 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1236: objectstore_tool: shift 2019-12-21T16:06:26.769 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1238: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING list-attrs 2019-12-21T16:06:26.769 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1205: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2019-12-21T16:06:26.769 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nowait: shift 2019-12-21T16:06:26.769 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1207: _objectstore_tool_nowait: local id=1 2019-12-21T16:06:26.770 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1208: _objectstore_tool_nowait: shift 2019-12-21T16:06:26.770 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1210: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2019-12-21T16:06:26.770 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2019-12-21T16:06:26.770 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2019-12-21T16:06:26.770 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2019-12-21T16:06:26.770 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2019-12-21T16:06:26.770 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2019-12-21T16:06:26.771 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2019-12-21T16:06:26.771 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1212: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING list-attrs 2019-12-21T16:06:26.771 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1193: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2019-12-21T16:06:26.771 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1194: _objectstore_tool_nodown: shift 2019-12-21T16:06:26.771 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1195: _objectstore_tool_nodown: local id=1 2019-12-21T16:06:26.771 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: _objectstore_tool_nodown: shift 2019-12-21T16:06:26.771 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2019-12-21T16:06:26.772 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING list-attrs 2019-12-21T16:06:26.772 INFO:tasks.workunit.client.0.smithi172.stderr:119575: _ 2019-12-21T16:06:26.772 INFO:tasks.workunit.client.0.smithi172.stderr:119575: hinfo_key 2019-12-21T16:06:26.772 INFO:tasks.workunit.client.0.smithi172.stderr:119575: snapset 2019-12-21T16:06:26.772 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1213: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2019-12-21T16:06:26.772 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:842: activate_osd: local dir=td/osd-scrub-repair 2019-12-21T16:06:26.772 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: shift 2019-12-21T16:06:26.773 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: local id=1 2019-12-21T16:06:26.773 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: shift 2019-12-21T16:06:26.773 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local osd_data=td/osd-scrub-repair/1 2019-12-21T16:06:26.773 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local 'ceph_args=--fsid=973e3fc1-e63e-4bde-9db6-84c68fd5e93e --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2019-12-21T16:06:26.773 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2019-12-21T16:06:26.774 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-journal-size=100' 2019-12-21T16:06:26.774 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2019-12-21T16:06:26.774 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2019-12-21T16:06:26.774 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2019-12-21T16:06:26.774 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --chdir=' 2019-12-21T16:06:26.774 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+= 2019-12-21T16:06:26.774 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2019-12-21T16:06:26.775 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: get_asok_path 2019-12-21T16:06:26.775 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2019-12-21T16:06:26.775 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2019-12-21T16:06:26.775 INFO:tasks.workunit.client.0.smithi172.stderr:119575: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2019-12-21T16:06:26.775 INFO:tasks.workunit.client.0.smithi172.stderr:119575: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T16:06:26.775 INFO:tasks.workunit.client.0.smithi172.stderr:119575: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.119575 2019-12-21T16:06:26.775 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.119575/$cluster-$name.asok' 2019-12-21T16:06:26.776 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.119575/$cluster-$name.asok' 2019-12-21T16:06:26.776 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --debug-osd=20' 2019-12-21T16:06:26.776 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2019-12-21T16:06:26.776 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2019-12-21T16:06:26.776 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2019-12-21T16:06:26.777 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2019-12-21T16:06:26.777 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2019-12-21T16:06:26.777 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' ' 2019-12-21T16:06:26.777 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+= 2019-12-21T16:06:26.777 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: mkdir -p td/osd-scrub-repair/1 2019-12-21T16:06:26.777 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: echo start osd.1 2019-12-21T16:06:26.777 INFO:tasks.workunit.client.0.smithi172.stderr:119575: start osd.1 2019-12-21T16:06:26.778 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph-osd -i 1 --fsid=973e3fc1-e63e-4bde-9db6-84c68fd5e93e --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.119575/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2019-12-21T16:06:26.778 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: cat td/osd-scrub-repair/1/whoami 2019-12-21T16:06:26.931 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: objectstore_tool: local dir=td/osd-scrub-repair 2019-12-21T16:06:26.932 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: objectstore_tool: shift 2019-12-21T16:06:26.932 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1235: objectstore_tool: local id=3 2019-12-21T16:06:26.932 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1236: objectstore_tool: shift 2019-12-21T16:06:26.932 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1238: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING list-attrs 2019-12-21T16:06:26.932 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1205: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2019-12-21T16:06:26.933 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nowait: shift 2019-12-21T16:06:26.933 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1207: _objectstore_tool_nowait: local id=3 2019-12-21T16:06:26.933 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1208: _objectstore_tool_nowait: shift 2019-12-21T16:06:26.933 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1210: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2019-12-21T16:06:26.933 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2019-12-21T16:06:26.933 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2019-12-21T16:06:26.933 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2019-12-21T16:06:26.934 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2019-12-21T16:06:26.934 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2019-12-21T16:06:26.934 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2019-12-21T16:06:26.934 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1212: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING list-attrs 2019-12-21T16:06:26.934 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1193: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2019-12-21T16:06:26.934 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1194: _objectstore_tool_nodown: shift 2019-12-21T16:06:26.935 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1195: _objectstore_tool_nodown: local id=3 2019-12-21T16:06:26.935 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: _objectstore_tool_nodown: shift 2019-12-21T16:06:26.935 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2019-12-21T16:06:26.935 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING list-attrs 2019-12-21T16:06:26.935 INFO:tasks.workunit.client.0.smithi172.stderr:119575: Error getting attr on : 2.0s0_head,0#-4:00000000:::scrub_2.0s0:head#, (61) No data available 2019-12-21T16:06:26.935 INFO:tasks.workunit.client.0.smithi172.stderr:119575: _ 2019-12-21T16:06:26.935 INFO:tasks.workunit.client.0.smithi172.stderr:119575: hinfo_key 2019-12-21T16:06:26.936 INFO:tasks.workunit.client.0.smithi172.stderr:119575: snapset 2019-12-21T16:06:26.936 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1213: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2019-12-21T16:06:26.936 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:842: activate_osd: local dir=td/osd-scrub-repair 2019-12-21T16:06:26.936 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: shift 2019-12-21T16:06:26.936 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: local id=3 2019-12-21T16:06:26.936 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: shift 2019-12-21T16:06:26.936 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local osd_data=td/osd-scrub-repair/3 2019-12-21T16:06:26.937 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local 'ceph_args=--fsid=973e3fc1-e63e-4bde-9db6-84c68fd5e93e --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2019-12-21T16:06:26.937 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2019-12-21T16:06:26.937 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-journal-size=100' 2019-12-21T16:06:26.937 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2019-12-21T16:06:26.938 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2019-12-21T16:06:26.938 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2019-12-21T16:06:26.938 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --chdir=' 2019-12-21T16:06:26.938 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+= 2019-12-21T16:06:26.938 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2019-12-21T16:06:26.938 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: get_asok_path 2019-12-21T16:06:26.938 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2019-12-21T16:06:26.939 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2019-12-21T16:06:26.939 INFO:tasks.workunit.client.0.smithi172.stderr:119575: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2019-12-21T16:06:26.939 INFO:tasks.workunit.client.0.smithi172.stderr:119575: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T16:06:26.939 INFO:tasks.workunit.client.0.smithi172.stderr:119575: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.119575 2019-12-21T16:06:26.939 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.119575/$cluster-$name.asok' 2019-12-21T16:06:26.939 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.119575/$cluster-$name.asok' 2019-12-21T16:06:26.939 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --debug-osd=20' 2019-12-21T16:06:26.940 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2019-12-21T16:06:26.940 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2019-12-21T16:06:26.940 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2019-12-21T16:06:26.940 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2019-12-21T16:06:26.941 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2019-12-21T16:06:26.941 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' ' 2019-12-21T16:06:26.941 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+= 2019-12-21T16:06:26.941 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: mkdir -p td/osd-scrub-repair/3 2019-12-21T16:06:26.941 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: echo start osd.3 2019-12-21T16:06:26.941 INFO:tasks.workunit.client.0.smithi172.stderr:119575: start osd.3 2019-12-21T16:06:26.941 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph-osd -i 3 --fsid=973e3fc1-e63e-4bde-9db6-84c68fd5e93e --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.119575/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2019-12-21T16:06:28.678 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers/qa/standalone/ceph-helpers.sh:871: activate_osd: '[' 1 = 1 ']' 2019-12-21T16:06:28.678 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph osd dump --format=json 2019-12-21T16:06:28.678 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: jq '.flags_set[]' 2019-12-21T16:06:28.678 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: grep -q '"noup"' 2019-12-21T16:06:28.678 INFO:tasks.workunit.client.0.smithi172.stderr:119575: 2019-12-21T16:06:26.788+0000 7f4d756a7ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T16:06:28.678 INFO:tasks.workunit.client.0.smithi172.stderr:119575: 2019-12-21T16:06:26.840+0000 7f4d756a7ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T16:06:28.679 INFO:tasks.workunit.client.0.smithi172.stderr:119575: 2019-12-21T16:06:26.841+0000 7f4d756a7ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T16:06:28.679 INFO:tasks.workunit.client.0.smithi172.stderr:119575: 2019-12-21T16:06:26.843+0000 7f4d756a7ec0 -1 Falling back to public interface 2019-12-21T16:06:28.679 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: wait_for_osd up 1 2019-12-21T16:06:28.679 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:914: wait_for_osd: local state=up 2019-12-21T16:06:28.679 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local id=1 2019-12-21T16:06:28.679 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: status=1 2019-12-21T16:06:28.679 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i=0 )) 2019-12-21T16:06:28.680 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i < 300 )) 2019-12-21T16:06:28.680 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: echo 0 2019-12-21T16:06:28.680 INFO:tasks.workunit.client.0.smithi172.stderr:119575: 0 2019-12-21T16:06:28.680 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: ceph osd dump 2019-12-21T16:06:28.680 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: grep 'osd.1 up' 2019-12-21T16:06:28.680 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: sleep 1 2019-12-21T16:06:28.681 INFO:tasks.workunit.client.0.smithi172.stderr:119575: 2019-12-21T16:06:27.480+0000 7f4d756a7ec0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2019-12-21T16:06:28.681 INFO:tasks.workunit.client.0.smithi172.stderr:119575: 2019-12-21T16:06:27.488+0000 7f4d756a7ec0 -1 journal do_read_entry(1241088): bad header magic 2019-12-21T16:06:28.681 INFO:tasks.workunit.client.0.smithi172.stderr:119575: 2019-12-21T16:06:27.488+0000 7f4d756a7ec0 -1 journal do_read_entry(1241088): bad header magic 2019-12-21T16:06:28.681 INFO:tasks.workunit.client.0.smithi172.stderr:119575: 2019-12-21T16:06:27.494+0000 7f4d756a7ec0 -1 osd.1 74 log_to_monitors {default=true} 2019-12-21T16:06:28.681 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i++ )) 2019-12-21T16:06:28.681 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i < 300 )) 2019-12-21T16:06:28.681 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: echo 1 2019-12-21T16:06:28.682 INFO:tasks.workunit.client.0.smithi172.stderr:119575: 1 2019-12-21T16:06:28.682 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: ceph osd dump 2019-12-21T16:06:28.682 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: grep 'osd.1 up' 2019-12-21T16:06:28.682 INFO:tasks.workunit.client.0.smithi172.stderr:119575: osd.1 up in weight 1 up_from 78 up_thru 78 down_at 75 last_clean_interval [72,74) [v2:127.0.0.1:6815/164021,v1:127.0.0.1:6817/164021] [v2:127.0.0.1:6819/164021,v1:127.0.0.1:6821/164021] exists,up 627d9f35-2c70-4ea0-8821-1ca4dc71496b 2019-12-21T16:06:28.682 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=0 2019-12-21T16:06:28.682 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: break 2019-12-21T16:06:28.682 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: return 0 2019-12-21T16:06:28.683 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1239: objectstore_tool: wait_for_clean 2019-12-21T16:06:28.683 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1548: wait_for_clean: local cmd= 2019-12-21T16:06:28.683 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1549: wait_for_clean: local num_active_clean=-1 2019-12-21T16:06:28.683 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1550: wait_for_clean: local cur_active_clean 2019-12-21T16:06:28.780 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1551: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_.sh:871: activate_osd: cat td/osd-scrub-repair/3/whoami 2019-12-21T16:06:28.781 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: '[' 3 = 3 ']' 2019-12-21T16:06:28.781 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph osd dump --format=json 2019-12-21T16:06:28.781 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: jq '.flags_set[]' 2019-12-21T16:06:28.781 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: grep -q '"noup"' 2019-12-21T16:06:28.781 INFO:tasks.workunit.client.0.smithi172.stderr:119575: 2019-12-21T16:06:26.951+0000 7f8f877c5ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T16:06:28.781 INFO:tasks.workunit.client.0.smithi172.stderr:119575: 2019-12-21T16:06:26.953+0000 7f8f877c5ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T16:06:28.782 INFO:tasks.workunit.client.0.smithi172.stderr:119575: 2019-12-21T16:06:26.955+0000 7f8f877c5ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T16:06:28.782 INFO:tasks.workunit.client.0.smithi172.stderr:119575: 2019-12-21T16:06:26.956+0000 7f8f877c5ec0 -1 Falling back to public interface 2019-12-21T16:06:28.782 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: wait_for_osd up 3 2019-12-21T16:06:28.782 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:914: wait_for_osd: local state=up 2019-12-21T16:06:28.782 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local id=3 2019-12-21T16:06:28.782 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: status=1 2019-12-21T16:06:28.783 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i=0 )) 2019-12-21T16:06:28.783 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i < 300 )) 2019-12-21T16:06:28.783 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: echo 0 2019-12-21T16:06:28.783 INFO:tasks.workunit.client.0.smithi172.stderr:119575: 0 2019-12-21T16:06:28.783 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: ceph osd dump 2019-12-21T16:06:28.783 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: grep 'osd.3 up' 2019-12-21T16:06:28.784 INFO:tasks.workunit.client.0.smithi172.stderr:119575: 2019-12-21T16:06:27.405+0000 7f8f877c5ec0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2019-12-21T16:06:28.784 INFO:tasks.workunit.client.0.smithi172.stderr:119575: 2019-12-21T16:06:27.413+0000 7f8f877c5ec0 -1 journal do_read_entry(1269760): bad header magic 2019-12-21T16:06:28.784 INFO:tasks.workunit.client.0.smithi172.stderr:119575: 2019-12-21T16:06:27.413+0000 7f8f877c5ec0 -1 journal do_read_entry(1269760): bad header magic 2019-12-21T16:06:28.784 INFO:tasks.workunit.client.0.smithi172.stderr:119575: 2019-12-21T16:06:27.421+0000 7f8f877c5ec0 -1 osd.3 74 log_to_monitors {default=true} 2019-12-21T16:06:28.784 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: sleep 1 2019-12-21T16:06:28.784 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i++ )) 2019-12-21T16:06:28.785 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i < 300 )) 2019-12-21T16:06:28.785 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: echo 1 2019-12-21T16:06:28.785 INFO:tasks.workunit.client.0.smithi172.stderr:119575: 1 2019-12-21T16:06:28.785 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: ceph osd dump 2019-12-21T16:06:28.785 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: grep 'osd.3 up' 2019-12-21T16:06:28.785 INFO:tasks.workunit.client.0.smithi172.stderr:119575: osd.3 up in weight 1 up_from 78 up_thru 78 down_at 75 last_clean_interval [72,74) [v2:127.0.0.1:6826/164074,v1:127.0.0.1:6827/164074] [v2:127.0.0.1:6828/164074,v1:127.0.0.1:6829/164074] exists,up d2b8f48d-5699-449e-8fc7-707402a64a19 2019-12-21T16:06:28.785 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=0 2019-12-21T16:06:28.785 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: break 2019-12-21T16:06:28.786 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: return 0 2019-12-21T16:06:28.786 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1239: objectstore_tool: wait_for_clean 2019-12-21T16:06:28.786 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1548: wait_for_clean: local cmd= 2019-12-21T16:06:28.786 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1549: wait_for_clean: local num_active_clean=-1 2019-12-21T16:06:28.786 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1550: wait_for_clean: local cur_active_clean 2019-12-21T16:06:29.522 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.CLEAN_TIMEOUT .1)) 2019-12-21T16:06:29.522 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1551: wait_for_clean: get_timeout_delays 90 .1 2019-12-21T16:06:29.523 INFO:tasks.workunit.client.0.smithi172.stderr:119575: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1494: get_timeout_delays: shopt -q -o xtrace 2019-12-21T16:06:29.523 INFO:tasks.workunit.client.0.smithi172.stderr:119575: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1494: get_timeout_delays: echo true 2019-12-21T16:06:29.523 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1494: get_timeout_delays: local trace=true 2019-12-21T16:06:29.523 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1495: get_timeout_delays: true 2019-12-21T16:06:29.523 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1495: get_timeout_delays: shopt -u -o xtrace 2019-12-21T16:06:29.524 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1551: wait_for_clean: local -a delays 2019-12-21T16:06:29.524 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1552: wait_for_clean: local -i loop=0 2019-12-21T16:06:29.524 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1554: wait_for_clean: flush_pg_stats 2019-12-21T16:06:29.524 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1977: flush_pg_stats: local timeout=300 2019-12-21T16:06:29.524 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1979: flush_pg_stats: ceph osd ls 2019-12-21T16:06:29.524 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1979: flush_pg_stats: ids='0 2019-12-21T16:06:29.524 INFO:tasks.workunit.client.0.smithi172.stderr:119575: 1 2019-12-21T16:06:29.525 INFO:tasks.workunit.client.0.smithi172.stderr:119575: 2 2019-12-21T16:06:29.525 INFO:tasks.workunit.client.0.smithi172.stderr:119575: 3' 2019-12-21T16:06:29.525 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1980: flush_pg_stats: seqs= 2019-12-21T16:06:29.525 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: flush_pg_stats: for osd in $ids 2019-12-21T16:06:29.525 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2019-12-21T16:06:29.525 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: seq=21474836519 2019-12-21T16:06:29.525 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: flush_pg_stats: seqs=' 0-21474836519' 2019-12-21T16:06:29.526 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: flush_pg_stats: for osd in $ids 2019-12-21T16:06:29.527 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2019-12-21T16:06:29.527 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: seq=335007449090 2019-12-21T16:06:29.528 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: flush_pg_stats: seqs=' 0-21474836519 1-335007449090' 2019-12-21T16:06:29.528 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: flush_pg_stats: for osd in $ids 2019-12-21T16:06:29.528 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2019-12-21T16:06:29.528 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: seq=292057776138 2019-12-21T16:06:29.528 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: flush_pg_stats: seqs=' 0-21474836519 1-335007449090 2-292057776138' 2019-12-21T16:06:29.528 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: flush_pg_stats: for osd in $ids 2019-12-21T16:06:29.529 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2019-12-21T16:06:29.529 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: seq=335007449090 2019-12-21T16:06:29.529 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: flush_pg_stats: seqs=' 0-21474836519 1-335007449090 2-292057776138 3-335007449090' 2019-12-21T16:06:29.529 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: flush_pg_stats: for s in $seqs 2019-12-21T16:06:29.529 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: echo 0-21474836519 2019-12-21T16:06:29.529 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: cut -d - -f 1 2019-12-21T16:06:29.530 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: osd=0 2019-12-21T16:06:29.566 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: echo 0client.0/qa/standalone/ceph-helpers.sh:1551: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2019-12-21T16:06:29.567 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1551: wait_for_clean: get_timeout_delays 90 .1 2019-12-21T16:06:29.567 INFO:tasks.workunit.client.0.smithi172.stderr:119575: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1494: get_timeout_delays: shopt -q -o xtrace 2019-12-21T16:06:29.567 INFO:tasks.workunit.client.0.smithi172.stderr:119575: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1494: get_timeout_delays: echo true 2019-12-21T16:06:29.568 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1494: get_timeout_delays: local trace=true 2019-12-21T16:06:29.568 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1495: get_timeout_delays: true 2019-12-21T16:06:29.568 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1495: get_timeout_delays: shopt -u -o xtrace 2019-12-21T16:06:29.569 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1551: wait_for_clean: local -a delays 2019-12-21T16:06:29.570 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1552: wait_for_clean: local -i loop=0 2019-12-21T16:06:29.570 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1554: wait_for_clean: flush_pg_stats 2019-12-21T16:06:29.570 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1977: flush_pg_stats: local timeout=300 2019-12-21T16:06:29.570 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1979: flush_pg_stats: ceph osd ls 2019-12-21T16:06:29.571 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1979: flush_pg_stats: ids='0 2019-12-21T16:06:29.571 INFO:tasks.workunit.client.0.smithi172.stderr:119575: 1 2019-12-21T16:06:29.571 INFO:tasks.workunit.client.0.smithi172.stderr:119575: 2 2019-12-21T16:06:29.571 INFO:tasks.workunit.client.0.smithi172.stderr:119575: 3' 2019-12-21T16:06:29.571 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1980: flush_pg_stats: seqs= 2019-12-21T16:06:29.572 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: flush_pg_stats: for osd in $ids 2019-12-21T16:06:29.572 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2019-12-21T16:06:29.572 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: seq=21474836520 2019-12-21T16:06:29.572 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: flush_pg_stats: seqs=' 0-21474836520' 2019-12-21T16:06:29.573 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: flush_pg_stats: for osd in $ids 2019-12-21T16:06:29.573 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2019-12-21T16:06:29.573 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: seq=335007449091 2019-12-21T16:06:29.573 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: flush_pg_stats: seqs=' 0-21474836520 1-335007449091' 2019-12-21T16:06:29.574 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: flush_pg_stats: for osd in $ids 2019-12-21T16:06:29.574 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2019-12-21T16:06:29.574 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: seq=292057776139 2019-12-21T16:06:29.574 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: flush_pg_stats: seqs=' 0-21474836520 1-335007449091 2-292057776139' 2019-12-21T16:06:29.575 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: flush_pg_stats: for osd in $ids 2019-12-21T16:06:29.575 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2019-12-21T16:06:29.575 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: seq=335007449091 2019-12-21T16:06:29.575 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: flush_pg_stats: seqs=' 0-21474836520 1-335007449091 2-292057776139 3-335007449091' 2019-12-21T16:06:29.576 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: flush_pg_stats: for s in $seqs 2019-12-21T16:06:29.576 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: echo 0-21474836520 2019-12-21T16:06:29.576 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: cut -d - -f 1 2019-12-21T16:06:29.576 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: osd=0 2019-12-21T16:06:32.857 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: echo 0-21474836520 2019-12-21T16:06:32.857 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: cut -d - -f 2 2019-12-21T16:06:32.858 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: seq=21474836520 2019-12-21T16:06:32.858 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1989: flush_pg_stats: echo 'waiting osd.0 seq 21474836520' 2019-12-21T16:06:32.858 INFO:tasks.workunit.client.0.smithi172.stderr:119575: waiting osd.0 seq 21474836520 2019-12-21T16:06:32.858 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 0 2019-12-21T16:06:32.858 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 21474836518 -lt 21474836520 2019-12-21T16:06:32.858 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1991: flush_pg_stats: sleep 1 2019-12-21T16:06:32.858 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1992: flush_pg_stats: '[' 300 -eq 0 ']' 2019-12-21T16:06:32.859 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 0 2019-12-21T16:06:32.859 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 21474836518 -lt 21474836520 2019-12-21T16:06:32.859 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1991: flush_pg_stats: sleep 1 2019-12-21T16:06:32.859 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1992: flush_pg_stats: '[' 299 -eq 0 ']' 2019-12-21T16:06:32.859 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 0 2019-12-21T16:06:32.859 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 21474836520 -lt 21474836520 2019-12-21T16:06:32.859 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: flush_pg_stats: for s in $seqs 2019-12-21T16:06:32.860 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: echo 1-335007449091 2019-12-21T16:06:32.860 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: cut -d - -f 1 2019-12-21T16:06:32.860 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: osd=1 2019-12-21T16:06:32.860 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: echo 1-335007449091 2019-12-21T16:06:32.860 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: cut -d - -f 2 2019-12-21T16:06:32.860 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: seq=335007449091 2019-12-21T16:06:32.860 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1989: flush_pg_stats: echo 'waiting osd.1 seq 335007449091' 2019-12-21T16:06:32.861 INFO:tasks.workunit.client.0.smithi172.stderr:119575: waiting osd.1 seq 335007449091 2019-12-21T16:06:32.861 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 1 2019-12-21T16:06:32.861 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 335007449091 -lt 335007449091 2019-12-21T16:06:32.861 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: flush_pg_stats: for s in $seqs 2019-12-21T16:06:32.861 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: echo 2-292057776139 2019-12-21T16:06:32.861 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: cut -d - -f 1 2019-12-21T16:06:32.861 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: osd=2 2019-12-21T16:06:32.862 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: echo 2-292057776139 2019-12-21T16:06:32.862 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: cut -d - -f 2 2019-12-21T16:06:32.862 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: seq=292057776139 2019-12-21T16:06:32.862 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1989: flush_pg_stats: echo 'waiting osd.2 seq 292057776139' 2019-12-21T16:06:32.862 INFO:tasks.workunit.client.0.smithi172.stderr:119575: waiting osd.2 seq 292057776139 2019-12-21T16:06:32.862 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 2 2019-12-21T16:06:32.863 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/st-21474836519 2019-12-21T16:06:32.863 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: cut -d - -f 2 2019-12-21T16:06:32.863 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: seq=21474836519 2019-12-21T16:06:32.863 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1989: flush_pg_stats: echo 'waiting osd.0 seq 21474836519' 2019-12-21T16:06:32.863 INFO:tasks.workunit.client.0.smithi172.stderr:119575: waiting osd.0 seq 21474836519 2019-12-21T16:06:32.863 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 0 2019-12-21T16:06:32.864 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 21474836518 -lt 21474836519 2019-12-21T16:06:32.864 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1991: flush_pg_stats: sleep 1 2019-12-21T16:06:32.864 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1992: flush_pg_stats: '[' 300 -eq 0 ']' 2019-12-21T16:06:32.864 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 0 2019-12-21T16:06:32.864 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 21474836518 -lt 21474836519 2019-12-21T16:06:32.864 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1991: flush_pg_stats: sleep 1 2019-12-21T16:06:32.864 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1992: flush_pg_stats: '[' 299 -eq 0 ']' 2019-12-21T16:06:32.865 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 0 2019-12-21T16:06:32.865 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 21474836520 -lt 21474836519 2019-12-21T16:06:32.865 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: flush_pg_stats: for s in $seqs 2019-12-21T16:06:32.865 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: echo 1-335007449090 2019-12-21T16:06:32.865 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: cut -d - -f 1 2019-12-21T16:06:32.866 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: osd=1 2019-12-21T16:06:32.866 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: echo 1-335007449090 2019-12-21T16:06:32.866 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: cut -d - -f 2 2019-12-21T16:06:32.866 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: seq=335007449090 2019-12-21T16:06:32.866 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1989: flush_pg_stats: echo 'waiting osd.1 seq 335007449090' 2019-12-21T16:06:32.866 INFO:tasks.workunit.client.0.smithi172.stderr:119575: waiting osd.1 seq 335007449090 2019-12-21T16:06:32.866 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 1 2019-12-21T16:06:32.867 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 335007449091 -lt 335007449090 2019-12-21T16:06:32.867 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: flush_pg_stats: for s in $seqs 2019-12-21T16:06:32.867 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: echo 2-292057776138 2019-12-21T16:06:32.867 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: cut -d - -f 1 2019-12-21T16:06:32.867 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: osd=2 2019-12-21T16:06:32.867 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: echo 2-292057776138 2019-12-21T16:06:32.867 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: cut -d - -f 2 2019-12-21T16:06:32.868 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: seq=292057776138 2019-12-21T16:06:32.868 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1989: flush_pg_stats: echo 'waiting osd.2 seq 292057776138' 2019-12-21T16:06:32.868 INFO:tasks.workunit.client.0.smithi172.stderr:119575: waiting osd.2 seq 292057776138 2019-12-21T16:06:32.869 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 2 2019-12-21T16:06:32.869 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 292057776139 -lt 292057776138 2019-12-21T16:06:33.926 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubunandalone/ceph-helpers.sh:1990: flush_pg_stats: test 292057776139 -lt 292057776139 2019-12-21T16:06:33.926 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: flush_pg_stats: for s in $seqs 2019-12-21T16:06:33.926 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: echo 3-335007449091 2019-12-21T16:06:33.927 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: cut -d - -f 1 2019-12-21T16:06:33.927 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: osd=3 2019-12-21T16:06:33.927 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: echo 3-335007449091 2019-12-21T16:06:33.927 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: cut -d - -f 2 2019-12-21T16:06:33.927 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: seq=335007449091 2019-12-21T16:06:33.927 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1989: flush_pg_stats: echo 'waiting osd.3 seq 335007449091' 2019-12-21T16:06:33.927 INFO:tasks.workunit.client.0.smithi172.stderr:119575: waiting osd.3 seq 335007449091 2019-12-21T16:06:33.928 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 3 2019-12-21T16:06:33.928 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 335007449091 -lt 335007449091 2019-12-21T16:06:33.928 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1555: wait_for_clean: get_num_pgs 2019-12-21T16:06:33.928 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: ceph --format json status 2019-12-21T16:06:33.928 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: jq .pgmap.num_pgs 2019-12-21T16:06:33.928 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1555: wait_for_clean: test 5 == 0 2019-12-21T16:06:33.928 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1559: wait_for_clean: true 2019-12-21T16:06:33.929 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1563: wait_for_clean: get_num_active_clean 2019-12-21T16:06:33.929 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: get_num_active_clean: local expression 2019-12-21T16:06:33.929 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2019-12-21T16:06:33.929 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("stale") | not)' 2019-12-21T16:06:33.929 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: ceph --format json pg dump pgs 2019-12-21T16:06:33.929 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2019-12-21T16:06:33.929 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1563: wait_for_clean: cur_active_clean=5 2019-12-21T16:06:33.930 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1564: wait_for_clean: get_num_pgs 2019-12-21T16:06:33.930 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: ceph --format json status 2019-12-21T16:06:33.930 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: jq .pgmap.num_pgs 2019-12-21T16:06:33.930 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1564: wait_for_clean: test 5 = 5 2019-12-21T16:06:33.930 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1564: wait_for_clean: break 2019-12-21T16:06:33.930 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: return 0 2019-12-21T16:06:33.930 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1911: run_in_background: return 0 2019-12-21T16:06:33.931 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1946: wait_background: for pid in $pids 2019-12-21T16:06:33.931 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1947: wait_background: wait 163888 2019-12-21T16:06:33.932 INFO:tasks.workunit.client.0.smithi172.stderr:tu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: flush_pg_stats: for s in $seqs 2019-12-21T16:06:33.932 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: echo 3-335007449090 2019-12-21T16:06:33.932 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: cut -d - -f 1 2019-12-21T16:06:33.932 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: osd=3 2019-12-21T16:06:33.932 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: echo 3-335007449090 2019-12-21T16:06:33.933 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: cut -d - -f 2 2019-12-21T16:06:33.933 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: seq=335007449090 2019-12-21T16:06:33.933 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1989: flush_pg_stats: echo 'waiting osd.3 seq 335007449090' 2019-12-21T16:06:33.933 INFO:tasks.workunit.client.0.smithi172.stderr:119575: waiting osd.3 seq 335007449090 2019-12-21T16:06:33.933 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 3 2019-12-21T16:06:33.933 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 335007449091 -lt 335007449090 2019-12-21T16:06:33.933 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1555: wait_for_clean: get_num_pgs 2019-12-21T16:06:33.934 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: ceph --format json status 2019-12-21T16:06:33.934 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: jq .pgmap.num_pgs 2019-12-21T16:06:33.934 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1555: wait_for_clean: test 5 == 0 2019-12-21T16:06:33.934 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1559: wait_for_clean: true 2019-12-21T16:06:33.934 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1563: wait_for_clean: get_num_active_clean 2019-12-21T16:06:33.934 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: get_num_active_clean: local expression 2019-12-21T16:06:33.934 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2019-12-21T16:06:33.935 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("stale") | not)' 2019-12-21T16:06:33.935 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: ceph --format json pg dump pgs 2019-12-21T16:06:33.935 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2019-12-21T16:06:33.935 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1563: wait_for_clean: cur_active_clean=5 2019-12-21T16:06:33.935 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1564: wait_for_clean: get_num_pgs 2019-12-21T16:06:33.935 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: ceph --format json status 2019-12-21T16:06:33.935 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: jq .pgmap.num_pgs 2019-12-21T16:06:33.936 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1564: wait_for_clean: test 5 = 5 2019-12-21T16:06:33.936 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1564: wait_for_clean: break 2019-12-21T16:06:33.936 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: return 0 2019-12-21T16:06:33.936 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1911: run_in_background: return 0 2019-12-21T16:06:33.936 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1954: wait_background: eval 'pids='\'''\''' 2019-12-21T16:06:33.936 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1954: wait_background: pids= 2019-12-21T16:06:33.937 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1956: wait_background: return 0 2019-12-21T16:06:33.937 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:218: corrupt_and_repair_two: return_code=0 2019-12-21T16:06:33.937 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:219: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2019-12-21T16:06:33.937 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:221: corrupt_and_repair_two: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2019-12-21T16:06:43.255 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:222: corrupt_and_repair_two: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2019-12-21T16:06:43.257 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:730: corrupt_and_repair_jerasure: teardown td/osd-scrub-repair 2019-12-21T16:06:43.257 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2019-12-21T16:06:43.257 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2019-12-21T16:06:43.257 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2019-12-21T16:06:43.257 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2019-12-21T16:06:43.258 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2019-12-21T16:06:43.258 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2019-12-21T16:06:43.258 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2019-12-21T16:06:43.259 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2019-12-21T16:06:43.370 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2019-12-21T16:06:43.370 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2019-12-21T16:06:43.371 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2019-12-21T16:06:43.371 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2019-12-21T16:06:43.372 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2019-12-21T16:06:43.372 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2019-12-21T16:06:43.373 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2019-12-21T16:06:43.374 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local 'pattern=|/usr/lib/systemd/systemd-coredump %P %u %g %s %t %c %h %e' 2019-12-21T16:06:43.374 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' '|' = '|' ']' 2019-12-21T16:06:43.375 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:177: teardown: pattern= 2019-12-21T16:06:43.375 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2019-12-21T16:06:43.375 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname '' 2019-12-21T16:06:43.376 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls . 2019-12-21T16:06:43.376 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2019-12-21T16:06:43.377 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2019-12-21T16:06:43.401 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2019-12-21T16:06:43.401 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T16:06:43.401 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.119575 2019-12-21T16:06:43.401 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.119575 2019-12-21T16:06:43.403 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2019-12-21T16:06:43.404 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2019-12-21T16:06:43.404 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:62: run: for func in $funcs 2019-12-21T16:06:43.404 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: TEST_corrupt_and_repair_jerasure_overwrites td/osd-scrub-repair 2019-12-21T16:06:43.404 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:738: TEST_corrupt_and_repair_jerasure_overwrites: '[' true = true ']' 2019-12-21T16:06:43.404 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:739: TEST_corrupt_and_repair_jerasure_overwrites: corrupt_and_repair_jerasure td/osd-scrub-repair true 2019-12-21T16:06:43.404 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:710: corrupt_and_repair_jerasure: local dir=td/osd-scrub-repair 2019-12-21T16:06:43.405 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:711: corrupt_and_repair_jerasure: local allow_overwrites=true 2019-12-21T16:06:43.405 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:712: corrupt_and_repair_jerasure: local poolname=ecpool 2019-12-21T16:06:43.405 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:714: corrupt_and_repair_jerasure: setup td/osd-scrub-repair 2019-12-21T16:06:43.405 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2019-12-21T16:06:43.405 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2019-12-21T16:06:43.405 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2019-12-21T16:06:43.405 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2019-12-21T16:06:43.406 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2019-12-21T16:06:43.406 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2019-12-21T16:06:43.406 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2019-12-21T16:06:43.406 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2019-12-21T16:06:43.406 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2019-12-21T16:06:43.406 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2019-12-21T16:06:43.407 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2019-12-21T16:06:43.407 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2019-12-21T16:06:43.407 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2019-12-21T16:06:43.408 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2019-12-21T16:06:43.408 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2019-12-21T16:06:43.408 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2019-12-21T16:06:43.408 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2019-12-21T16:06:43.409 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local 'pattern=|/usr/lib/systemd/systemd-coredump %P %u %g %s %t %c %h %e' 2019-12-21T16:06:43.410 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' '|' = '|' ']' 2019-12-21T16:06:43.410 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:177: teardown: pattern= 2019-12-21T16:06:43.410 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2019-12-21T16:06:43.410 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname '' 2019-12-21T16:06:43.411 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls . 2019-12-21T16:06:43.411 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2019-12-21T16:06:43.412 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2019-12-21T16:06:43.413 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2019-12-21T16:06:43.413 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T16:06:43.413 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.119575 2019-12-21T16:06:43.413 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.119575 2019-12-21T16:06:43.414 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2019-12-21T16:06:43.414 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2019-12-21T16:06:43.414 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2019-12-21T16:06:43.415 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2019-12-21T16:06:43.415 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T16:06:43.416 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.119575 2019-12-21T16:06:43.416 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.119575 2019-12-21T16:06:43.416 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2019-12-21T16:06:43.418 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2019-12-21T16:06:43.418 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2019-12-21T16:06:43.418 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2019-12-21T16:06:43.418 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:715: corrupt_and_repair_jerasure: run_mon td/osd-scrub-repair a 2019-12-21T16:06:43.418 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2019-12-21T16:06:43.418 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2019-12-21T16:06:43.419 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2019-12-21T16:06:43.419 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2019-12-21T16:06:43.419 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2019-12-21T16:06:43.419 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2019-12-21T16:06:43.710 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2019-12-21T16:06:43.710 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2019-12-21T16:06:43.711 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2019-12-21T16:06:43.711 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2019-12-21T16:06:43.711 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T16:06:43.711 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.119575 2019-12-21T16:06:43.711 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.119575/$cluster-$name.asok' 2019-12-21T16:06:43.712 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.119575/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 2019-12-21T16:06:43.894 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: cat 2019-12-21T16:06:43.895 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: get_config mon a fsid 2019-12-21T16:06:43.895 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: get_config: local daemon=mon 2019-12-21T16:06:43.895 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: get_config: local id=a 2019-12-21T16:06:43.896 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local config=fsid 2019-12-21T16:06:43.896 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: jq -r .fsid 2019-12-21T16:06:43.897 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: get_asok_path mon.a 2019-12-21T16:06:43.897 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2019-12-21T16:06:43.897 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2019-12-21T16:06:43.897 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T16:06:43.898 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T16:06:43.898 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.119575 2019-12-21T16:06:43.898 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.119575/ceph-mon.a.asok 2019-12-21T16:06:43.898 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: CEPH_ARGS= 2019-12-21T16:06:43.898 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: ceph --format json daemon /tmp/ceph-asok.119575/ceph-mon.a.asok config get fsid 2019-12-21T16:06:43.986 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: get_config mon a mon_host 2019-12-21T16:06:43.986 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: get_config: local daemon=mon 2019-12-21T16:06:43.986 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: get_config: local id=a 2019-12-21T16:06:43.986 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local config=mon_host 2019-12-21T16:06:43.987 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: jq -r .mon_host 2019-12-21T16:06:43.987 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: get_asok_path mon.a 2019-12-21T16:06:43.987 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2019-12-21T16:06:43.987 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2019-12-21T16:06:43.988 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2019-12-21T16:06:43.988 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T16:06:43.988 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.119575 2019-12-21T16:06:43.988 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.119575/ceph-mon.a.asok 2019-12-21T16:06:43.989 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: CEPH_ARGS= 2019-12-21T16:06:43.989 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: ceph --format json daemon /tmp/ceph-asok.119575/ceph-mon.a.asok config get mon_host 2019-12-21T16:06:44.074 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:716: corrupt_and_repair_jerasure: run_mgr td/osd-scrub-repair x 2019-12-21T16:06:44.074 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:551: run_mgr: local dir=td/osd-scrub-repair 2019-12-21T16:06:44.074 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: shift 2019-12-21T16:06:44.074 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local id=x 2019-12-21T16:06:44.074 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2019-12-21T16:06:44.074 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local data=td/osd-scrub-repair/x 2019-12-21T16:06:44.076 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:558: run_mgr: get_asok_path 2019-12-21T16:06:44.076 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2019-12-21T16:06:44.076 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2019-12-21T16:06:44.076 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2019-12-21T16:06:44.076 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T16:06:44.076 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.119575 2019-12-21T16:06:44.077 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.119575/$cluster-$name.asok' 2019-12-21T16:06:44.077 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:558: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2019-12-21T16:06:44.077 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:558: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.119575/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2019-12-21T16:06:44.296 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:717: corrupt_and_repair_jerasure: seq 0 3 2019-12-21T16:06:44.300 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:717: corrupt_and_repair_jerasure: for id in $(seq 0 3) 2019-12-21T16:06:44.300 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:718: corrupt_and_repair_jerasure: '[' true = true ']' 2019-12-21T16:06:44.300 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:719: corrupt_and_repair_jerasure: run_osd td/osd-scrub-repair 0 2019-12-21T16:06:44.300 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local dir=td/osd-scrub-repair 2019-12-21T16:06:44.301 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: shift 2019-12-21T16:06:44.301 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local id=0 2019-12-21T16:06:44.301 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2019-12-21T16:06:44.301 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local osd_data=td/osd-scrub-repair/0 2019-12-21T16:06:44.301 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local 'ceph_args=--fsid=973e3fc1-e63e-4bde-9db6-84c68fd5e93e --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2019-12-21T16:06:44.301 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2019-12-21T16:06:44.302 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-journal-size=100' 2019-12-21T16:06:44.302 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2019-12-21T16:06:44.302 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2019-12-21T16:06:44.302 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2019-12-21T16:06:44.302 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --chdir=' 2019-12-21T16:06:44.302 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+= 2019-12-21T16:06:44.302 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2019-12-21T16:06:44.303 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: get_asok_path 2019-12-21T16:06:44.303 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2019-12-21T16:06:44.303 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2019-12-21T16:06:44.303 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2019-12-21T16:06:44.303 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T16:06:44.303 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.119575 2019-12-21T16:06:44.303 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.119575/$cluster-$name.asok' 2019-12-21T16:06:44.304 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.119575/$cluster-$name.asok' 2019-12-21T16:06:44.304 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --debug-osd=20' 2019-12-21T16:06:44.304 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-ms=1' 2019-12-21T16:06:44.304 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-monc=20' 2019-12-21T16:06:44.305 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2019-12-21T16:06:44.305 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2019-12-21T16:06:44.305 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2019-12-21T16:06:44.305 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2019-12-21T16:06:44.305 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2019-12-21T16:06:44.305 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' ' 2019-12-21T16:06:44.306 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+= 2019-12-21T16:06:44.306 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: mkdir -p td/osd-scrub-repair/0 2019-12-21T16:06:44.306 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: uuidgen 2019-12-21T16:06:44.308 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: local uuid=c5f48369-6f98-4835-9ac3-881fec2bb8e4 2019-12-21T16:06:44.308 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: echo 'add osd0 c5f48369-6f98-4835-9ac3-881fec2bb8e4' 2019-12-21T16:06:44.308 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: ceph-authtool --gen-print-key 2019-12-21T16:06:44.309 INFO:tasks.workunit.client.0.smithi172.stdout:add osd0 c5f48369-6f98-4835-9ac3-881fec2bb8e4 2019-12-21T16:06:44.314 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: OSD_SECRET=AQAUQ/5dHAuaEhAAgKIMaolCaAQN3ldIxXMfMQ== 2019-12-21T16:06:44.314 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo '{"cephx_secret": "AQAUQ/5dHAuaEhAAgKIMaolCaAQN3ldIxXMfMQ=="}' 2019-12-21T16:06:44.314 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph osd new c5f48369-6f98-4835-9ac3-881fec2bb8e4 -i td/osd-scrub-repair/0/new.json 2019-12-21T16:06:44.593 INFO:tasks.workunit.client.0.smithi172.stdout:0 2019-12-21T16:06:44.603 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: rm td/osd-scrub-repair/0/new.json 2019-12-21T16:06:44.604 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-osd -i 0 --fsid=973e3fc1-e63e-4bde-9db6-84c68fd5e93e --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.119575/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAUQ/5dHAuaEhAAgKIMaolCaAQN3ldIxXMfMQ== --osd-uuid c5f48369-6f98-4835-9ac3-881fec2bb8e4 2019-12-21T16:06:44.625 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:06:44.622+0000 7f86ce833ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T16:06:44.627 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:06:44.624+0000 7f86ce833ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T16:06:44.628 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:06:44.626+0000 7f86ce833ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T16:06:44.629 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:06:44.626+0000 7f86ce833ec0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2019-12-21T16:06:44.629 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:06:44.626+0000 7f86ce833ec0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2019-12-21T16:06:44.629 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:06:44.626+0000 7f86ce833ec0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2019-12-21T16:06:44.677 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:06:44.674+0000 7f86ce833ec0 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2019-12-21T16:06:47.674 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2019-12-21T16:06:47.674 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: cat 2019-12-21T16:06:47.675 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: echo adding osd0 key to auth repository 2019-12-21T16:06:47.675 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2019-12-21T16:06:47.675 INFO:tasks.workunit.client.0.smithi172.stdout:adding osd0 key to auth repository 2019-12-21T16:06:47.865 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo start osd.0 2019-12-21T16:06:47.866 INFO:tasks.workunit.client.0.smithi172.stdout:start osd.0 2019-12-21T16:06:47.867 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph osd dump --format=json 2019-12-21T16:06:47.867 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph-osd -i 0 --fsid=973e3fc1-e63e-4bde-9db6-84c68fd5e93e --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.119575/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2019-12-21T16:06:47.867 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: jq '.flags_set[]' 2019-12-21T16:06:47.867 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: grep -q '"noup"' 2019-12-21T16:06:47.885 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:06:47.882+0000 7fca7c6cbec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T16:06:47.886 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:06:47.884+0000 7fca7c6cbec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T16:06:47.888 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:06:47.885+0000 7fca7c6cbec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T16:06:48.108 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: wait_for_osd up 0 2019-12-21T16:06:48.108 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:914: wait_for_osd: local state=up 2019-12-21T16:06:48.109 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local id=0 2019-12-21T16:06:48.109 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: status=1 2019-12-21T16:06:48.109 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i=0 )) 2019-12-21T16:06:48.109 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i < 300 )) 2019-12-21T16:06:48.109 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: echo 0 2019-12-21T16:06:48.109 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: ceph osd dump 2019-12-21T16:06:48.110 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: grep 'osd.0 up' 2019-12-21T16:06:48.110 INFO:tasks.workunit.client.0.smithi172.stdout:0 2019-12-21T16:06:48.334 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: sleep 1 2019-12-21T16:06:48.407 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:06:48.405+0000 7fca7c6cbec0 -1 Falling back to public interface 2019-12-21T16:06:49.134 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:06:49.132+0000 7fca7c6cbec0 -1 osd.0 0 log_to_monitors {default=true} 2019-12-21T16:06:49.336 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i++ )) 2019-12-21T16:06:49.336 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i < 300 )) 2019-12-21T16:06:49.336 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: echo 1 2019-12-21T16:06:49.336 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: ceph osd dump 2019-12-21T16:06:49.336 INFO:tasks.workunit.client.0.smithi172.stdout:1 2019-12-21T16:06:49.337 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: grep 'osd.0 up' 2019-12-21T16:06:49.606 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: sleep 1 2019-12-21T16:06:50.608 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i++ )) 2019-12-21T16:06:50.608 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i < 300 )) 2019-12-21T16:06:50.608 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: echo 2 2019-12-21T16:06:50.608 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: ceph osd dump 2019-12-21T16:06:50.608 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: grep 'osd.0 up' 2019-12-21T16:06:50.609 INFO:tasks.workunit.client.0.smithi172.stdout:2 2019-12-21T16:06:50.840 INFO:tasks.workunit.client.0.smithi172.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/165754,v1:127.0.0.1:6803/165754] [v2:127.0.0.1:6804/165754,v1:127.0.0.1:6805/165754] exists,up c5f48369-6f98-4835-9ac3-881fec2bb8e4 2019-12-21T16:06:50.840 INFO:tasks.workunit.client.0.smithi172.stdout:add osd1 356c69ab-d874-4bb4-813b-1608e502f8c0 2019-12-21T16:06:50.841 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=0 2019-12-21T16:06:50.841 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: break 2019-12-21T16:06:50.841 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: return 0 2019-12-21T16:06:50.841 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:717: corrupt_and_repair_jerasure: for id in $(seq 0 3) 2019-12-21T16:06:50.841 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:718: corrupt_and_repair_jerasure: '[' true = true ']' 2019-12-21T16:06:50.841 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:719: corrupt_and_repair_jerasure: run_osd td/osd-scrub-repair 1 2019-12-21T16:06:50.842 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local dir=td/osd-scrub-repair 2019-12-21T16:06:50.842 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: shift 2019-12-21T16:06:50.842 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local id=1 2019-12-21T16:06:50.842 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2019-12-21T16:06:50.842 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local osd_data=td/osd-scrub-repair/1 2019-12-21T16:06:50.842 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local 'ceph_args=--fsid=973e3fc1-e63e-4bde-9db6-84c68fd5e93e --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2019-12-21T16:06:50.843 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2019-12-21T16:06:50.843 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-journal-size=100' 2019-12-21T16:06:50.843 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2019-12-21T16:06:50.843 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2019-12-21T16:06:50.843 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2019-12-21T16:06:50.843 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --chdir=' 2019-12-21T16:06:50.843 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+= 2019-12-21T16:06:50.844 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2019-12-21T16:06:50.844 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: get_asok_path 2019-12-21T16:06:50.844 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2019-12-21T16:06:50.844 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2019-12-21T16:06:50.844 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2019-12-21T16:06:50.844 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T16:06:50.845 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.119575 2019-12-21T16:06:50.845 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.119575/$cluster-$name.asok' 2019-12-21T16:06:50.845 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.119575/$cluster-$name.asok' 2019-12-21T16:06:50.845 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --debug-osd=20' 2019-12-21T16:06:50.845 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-ms=1' 2019-12-21T16:06:50.845 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-monc=20' 2019-12-21T16:06:50.845 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2019-12-21T16:06:50.846 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2019-12-21T16:06:50.846 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2019-12-21T16:06:50.846 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2019-12-21T16:06:50.846 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2019-12-21T16:06:50.846 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' ' 2019-12-21T16:06:50.846 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+= 2019-12-21T16:06:50.846 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: mkdir -p td/osd-scrub-repair/1 2019-12-21T16:06:50.847 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: uuidgen 2019-12-21T16:06:50.847 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: local uuid=356c69ab-d874-4bb4-813b-1608e502f8c0 2019-12-21T16:06:50.847 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: echo 'add osd1 356c69ab-d874-4bb4-813b-1608e502f8c0' 2019-12-21T16:06:50.847 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: ceph-authtool --gen-print-key 2019-12-21T16:06:50.853 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: OSD_SECRET=AQAaQ/5dxwu7MhAAs176mKORE/gHEw97LI47kg== 2019-12-21T16:06:50.853 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo '{"cephx_secret": "AQAaQ/5dxwu7MhAAs176mKORE/gHEw97LI47kg=="}' 2019-12-21T16:06:50.853 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph osd new 356c69ab-d874-4bb4-813b-1608e502f8c0 -i td/osd-scrub-repair/1/new.json 2019-12-21T16:06:51.174 INFO:tasks.workunit.client.0.smithi172.stdout:1 2019-12-21T16:06:51.184 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: rm td/osd-scrub-repair/1/new.json 2019-12-21T16:06:51.185 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-osd -i 1 --fsid=973e3fc1-e63e-4bde-9db6-84c68fd5e93e --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.119575/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAaQ/5dxwu7MhAAs176mKORE/gHEw97LI47kg== --osd-uuid 356c69ab-d874-4bb4-813b-1608e502f8c0 2019-12-21T16:06:51.206 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:06:51.205+0000 7f771abfbec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T16:06:51.208 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:06:51.206+0000 7f771abfbec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T16:06:51.210 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:06:51.208+0000 7f771abfbec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T16:06:51.210 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:06:51.208+0000 7f771abfbec0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2019-12-21T16:06:51.210 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:06:51.208+0000 7f771abfbec0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2019-12-21T16:06:51.210 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:06:51.208+0000 7f771abfbec0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2019-12-21T16:06:51.274 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:06:51.272+0000 7f771abfbec0 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2019-12-21T16:06:54.274 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2019-12-21T16:06:54.274 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: cat 2019-12-21T16:06:54.275 INFO:tasks.workunit.client.0.smithi172.stdout:adding osd1 key to auth repository 2019-12-21T16:06:54.275 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: echo adding osd1 key to auth repository 2019-12-21T16:06:54.275 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2019-12-21T16:06:54.563 INFO:tasks.workunit.client.0.smithi172.stdout:start osd.1 2019-12-21T16:06:54.563 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo start osd.1 2019-12-21T16:06:54.563 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph-osd -i 1 --fsid=973e3fc1-e63e-4bde-9db6-84c68fd5e93e --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.119575/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2019-12-21T16:06:54.563 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph osd dump --format=json 2019-12-21T16:06:54.563 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: jq '.flags_set[]' 2019-12-21T16:06:54.564 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: grep -q '"noup"' 2019-12-21T16:06:54.582 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:06:54.580+0000 7f391e361ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T16:06:54.583 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:06:54.582+0000 7f391e361ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T16:06:54.584 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:06:54.583+0000 7f391e361ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T16:06:54.792 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: wait_for_osd up 1 2019-12-21T16:06:54.792 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:914: wait_for_osd: local state=up 2019-12-21T16:06:54.792 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local id=1 2019-12-21T16:06:54.792 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: status=1 2019-12-21T16:06:54.793 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i=0 )) 2019-12-21T16:06:54.793 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i < 300 )) 2019-12-21T16:06:54.795 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: echo 0 2019-12-21T16:06:54.796 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: ceph osd dump 2019-12-21T16:06:54.796 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: grep 'osd.1 up' 2019-12-21T16:06:54.796 INFO:tasks.workunit.client.0.smithi172.stdout:0 2019-12-21T16:06:55.020 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: sleep 1 2019-12-21T16:06:55.105 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:06:55.103+0000 7f391e361ec0 -1 Falling back to public interface 2019-12-21T16:06:55.872 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:06:55.870+0000 7f391e361ec0 -1 osd.1 0 log_to_monitors {default=true} 2019-12-21T16:06:56.021 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i++ )) 2019-12-21T16:06:56.022 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i < 300 )) 2019-12-21T16:06:56.022 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: echo 1 2019-12-21T16:06:56.022 INFO:tasks.workunit.client.0.smithi172.stdout:1 2019-12-21T16:06:56.022 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: grep 'osd.1 up' 2019-12-21T16:06:56.022 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: ceph osd dump 2019-12-21T16:06:56.255 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: sleep 1 2019-12-21T16:06:57.257 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i++ )) 2019-12-21T16:06:57.257 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i < 300 )) 2019-12-21T16:06:57.258 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: echo 2 2019-12-21T16:06:57.258 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: ceph osd dump 2019-12-21T16:06:57.259 INFO:tasks.workunit.client.0.smithi172.stdout:2 2019-12-21T16:06:57.259 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: grep 'osd.1 up' 2019-12-21T16:06:57.492 INFO:tasks.workunit.client.0.smithi172.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/166131,v1:127.0.0.1:6811/166131] [v2:127.0.0.1:6812/166131,v1:127.0.0.1:6813/166131] exists,up 356c69ab-d874-4bb4-813b-1608e502f8c0 2019-12-21T16:06:57.492 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=0 2019-12-21T16:06:57.492 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: break 2019-12-21T16:06:57.493 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: return 0 2019-12-21T16:06:57.493 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:717: corrupt_and_repair_jerasure: for id in $(seq 0 3) 2019-12-21T16:06:57.493 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:718: corrupt_and_repair_jerasure: '[' true = true ']' 2019-12-21T16:06:57.493 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:719: corrupt_and_repair_jerasure: run_osd td/osd-scrub-repair 2 2019-12-21T16:06:57.493 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local dir=td/osd-scrub-repair 2019-12-21T16:06:57.493 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: shift 2019-12-21T16:06:57.494 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local id=2 2019-12-21T16:06:57.494 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2019-12-21T16:06:57.494 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local osd_data=td/osd-scrub-repair/2 2019-12-21T16:06:57.494 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local 'ceph_args=--fsid=973e3fc1-e63e-4bde-9db6-84c68fd5e93e --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2019-12-21T16:06:57.494 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2019-12-21T16:06:57.494 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-journal-size=100' 2019-12-21T16:06:57.494 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2019-12-21T16:06:57.495 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2019-12-21T16:06:57.495 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2019-12-21T16:06:57.496 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --chdir=' 2019-12-21T16:06:57.496 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+= 2019-12-21T16:06:57.496 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2019-12-21T16:06:57.496 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: get_asok_path 2019-12-21T16:06:57.496 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2019-12-21T16:06:57.496 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2019-12-21T16:06:57.497 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2019-12-21T16:06:57.497 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T16:06:57.497 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.119575 2019-12-21T16:06:57.497 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.119575/$cluster-$name.asok' 2019-12-21T16:06:57.497 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.119575/$cluster-$name.asok' 2019-12-21T16:06:57.497 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --debug-osd=20' 2019-12-21T16:06:57.498 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-ms=1' 2019-12-21T16:06:57.498 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-monc=20' 2019-12-21T16:06:57.498 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2019-12-21T16:06:57.498 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2019-12-21T16:06:57.498 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2019-12-21T16:06:57.498 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2019-12-21T16:06:57.498 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2019-12-21T16:06:57.498 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' ' 2019-12-21T16:06:57.499 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+= 2019-12-21T16:06:57.499 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: mkdir -p td/osd-scrub-repair/2 2019-12-21T16:06:57.499 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: uuidgen 2019-12-21T16:06:57.500 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: local uuid=16235db4-ed2c-4f34-945d-b07522382896 2019-12-21T16:06:57.500 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: echo 'add osd2 16235db4-ed2c-4f34-945d-b07522382896' 2019-12-21T16:06:57.500 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: ceph-authtool --gen-print-key 2019-12-21T16:06:57.500 INFO:tasks.workunit.client.0.smithi172.stdout:add osd2 16235db4-ed2c-4f34-945d-b07522382896 2019-12-21T16:06:57.505 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: OSD_SECRET=AQAhQ/5d2ukEHhAAMcI8ol9TZ/voABBAMsraRQ== 2019-12-21T16:06:57.506 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo '{"cephx_secret": "AQAhQ/5d2ukEHhAAMcI8ol9TZ/voABBAMsraRQ=="}' 2019-12-21T16:06:57.506 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph osd new 16235db4-ed2c-4f34-945d-b07522382896 -i td/osd-scrub-repair/2/new.json 2019-12-21T16:06:57.821 INFO:tasks.workunit.client.0.smithi172.stdout:2 2019-12-21T16:06:57.831 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: rm td/osd-scrub-repair/2/new.json 2019-12-21T16:06:57.832 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-osd -i 2 --fsid=973e3fc1-e63e-4bde-9db6-84c68fd5e93e --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.119575/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAhQ/5d2ukEHhAAMcI8ol9TZ/voABBAMsraRQ== --osd-uuid 16235db4-ed2c-4f34-945d-b07522382896 2019-12-21T16:06:57.854 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:06:57.852+0000 7f7643010ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T16:06:57.855 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:06:57.854+0000 7f7643010ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T16:06:57.856 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:06:57.854+0000 7f7643010ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T16:06:57.857 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:06:57.855+0000 7f7643010ec0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2019-12-21T16:06:57.857 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:06:57.855+0000 7f7643010ec0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2019-12-21T16:06:57.857 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:06:57.855+0000 7f7643010ec0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2019-12-21T16:06:57.956 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:06:57.952+0000 7f7643010ec0 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2019-12-21T16:07:00.954 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2019-12-21T16:07:00.955 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: cat 2019-12-21T16:07:00.956 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: echo adding osd2 key to auth repository 2019-12-21T16:07:00.956 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2019-12-21T16:07:00.956 INFO:tasks.workunit.client.0.smithi172.stdout:adding osd2 key to auth repository 2019-12-21T16:07:01.238 INFO:tasks.workunit.client.0.smithi172.stdout:start osd.2 2019-12-21T16:07:01.238 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo start osd.2 2019-12-21T16:07:01.238 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph-osd -i 2 --fsid=973e3fc1-e63e-4bde-9db6-84c68fd5e93e --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.119575/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2019-12-21T16:07:01.239 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph osd dump --format=json 2019-12-21T16:07:01.239 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: jq '.flags_set[]' 2019-12-21T16:07:01.239 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: grep -q '"noup"' 2019-12-21T16:07:01.257 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:07:01.256+0000 7f1e2b885ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T16:07:01.259 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:07:01.258+0000 7f1e2b885ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T16:07:01.261 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:07:01.259+0000 7f1e2b885ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T16:07:01.471 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: wait_for_osd up 2 2019-12-21T16:07:01.472 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:914: wait_for_osd: local state=up 2019-12-21T16:07:01.472 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local id=2 2019-12-21T16:07:01.472 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: status=1 2019-12-21T16:07:01.472 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i=0 )) 2019-12-21T16:07:01.472 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i < 300 )) 2019-12-21T16:07:01.472 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: echo 0 2019-12-21T16:07:01.472 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: ceph osd dump 2019-12-21T16:07:01.473 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: grep 'osd.2 up' 2019-12-21T16:07:01.473 INFO:tasks.workunit.client.0.smithi172.stdout:0 2019-12-21T16:07:01.697 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: sleep 1 2019-12-21T16:07:01.777 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:07:01.776+0000 7f1e2b885ec0 -1 Falling back to public interface 2019-12-21T16:07:02.528 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:07:02.526+0000 7f1e2b885ec0 -1 osd.2 0 log_to_monitors {default=true} 2019-12-21T16:07:02.698 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i++ )) 2019-12-21T16:07:02.699 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i < 300 )) 2019-12-21T16:07:02.699 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: echo 1 2019-12-21T16:07:02.699 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: ceph osd dump 2019-12-21T16:07:02.699 INFO:tasks.workunit.client.0.smithi172.stdout:1 2019-12-21T16:07:02.700 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: grep 'osd.2 up' 2019-12-21T16:07:03.059 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: sleep 1 2019-12-21T16:07:04.060 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i++ )) 2019-12-21T16:07:04.061 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i < 300 )) 2019-12-21T16:07:04.061 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: echo 2 2019-12-21T16:07:04.061 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: ceph osd dump 2019-12-21T16:07:04.061 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: grep 'osd.2 up' 2019-12-21T16:07:04.061 INFO:tasks.workunit.client.0.smithi172.stdout:2 2019-12-21T16:07:04.299 INFO:tasks.workunit.client.0.smithi172.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/166508,v1:127.0.0.1:6819/166508] [v2:127.0.0.1:6820/166508,v1:127.0.0.1:6821/166508] exists,up 16235db4-ed2c-4f34-945d-b07522382896 2019-12-21T16:07:04.300 INFO:tasks.workunit.client.0.smithi172.stdout:add osd3 245349d5-5742-4af0-b9e3-1b7e828df542 2019-12-21T16:07:04.300 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=0 2019-12-21T16:07:04.300 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: break 2019-12-21T16:07:04.301 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: return 0 2019-12-21T16:07:04.301 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:717: corrupt_and_repair_jerasure: for id in $(seq 0 3) 2019-12-21T16:07:04.301 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:718: corrupt_and_repair_jerasure: '[' true = true ']' 2019-12-21T16:07:04.301 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:719: corrupt_and_repair_jerasure: run_osd td/osd-scrub-repair 3 2019-12-21T16:07:04.301 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local dir=td/osd-scrub-repair 2019-12-21T16:07:04.302 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: shift 2019-12-21T16:07:04.302 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local id=3 2019-12-21T16:07:04.302 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2019-12-21T16:07:04.302 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local osd_data=td/osd-scrub-repair/3 2019-12-21T16:07:04.302 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local 'ceph_args=--fsid=973e3fc1-e63e-4bde-9db6-84c68fd5e93e --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2019-12-21T16:07:04.302 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2019-12-21T16:07:04.302 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-journal-size=100' 2019-12-21T16:07:04.303 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2019-12-21T16:07:04.303 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2019-12-21T16:07:04.303 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2019-12-21T16:07:04.303 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --chdir=' 2019-12-21T16:07:04.303 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+= 2019-12-21T16:07:04.303 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2019-12-21T16:07:04.303 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: get_asok_path 2019-12-21T16:07:04.304 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2019-12-21T16:07:04.304 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2019-12-21T16:07:04.304 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2019-12-21T16:07:04.304 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T16:07:04.304 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.119575 2019-12-21T16:07:04.304 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.119575/$cluster-$name.asok' 2019-12-21T16:07:04.305 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.119575/$cluster-$name.asok' 2019-12-21T16:07:04.305 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --debug-osd=20' 2019-12-21T16:07:04.305 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-ms=1' 2019-12-21T16:07:04.305 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-monc=20' 2019-12-21T16:07:04.305 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2019-12-21T16:07:04.305 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2019-12-21T16:07:04.305 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2019-12-21T16:07:04.305 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2019-12-21T16:07:04.306 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2019-12-21T16:07:04.306 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' ' 2019-12-21T16:07:04.306 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+= 2019-12-21T16:07:04.306 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: mkdir -p td/osd-scrub-repair/3 2019-12-21T16:07:04.306 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: uuidgen 2019-12-21T16:07:04.306 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: local uuid=245349d5-5742-4af0-b9e3-1b7e828df542 2019-12-21T16:07:04.307 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: echo 'add osd3 245349d5-5742-4af0-b9e3-1b7e828df542' 2019-12-21T16:07:04.307 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: ceph-authtool --gen-print-key 2019-12-21T16:07:04.311 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: OSD_SECRET=AQAoQ/5dCW1wEhAALGqant2enfyib3KiqrPiyw== 2019-12-21T16:07:04.311 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo '{"cephx_secret": "AQAoQ/5dCW1wEhAALGqant2enfyib3KiqrPiyw=="}' 2019-12-21T16:07:04.311 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph osd new 245349d5-5742-4af0-b9e3-1b7e828df542 -i td/osd-scrub-repair/3/new.json 2019-12-21T16:07:04.627 INFO:tasks.workunit.client.0.smithi172.stdout:3 2019-12-21T16:07:04.637 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: rm td/osd-scrub-repair/3/new.json 2019-12-21T16:07:04.638 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-osd -i 3 --fsid=973e3fc1-e63e-4bde-9db6-84c68fd5e93e --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.119575/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAoQ/5dCW1wEhAALGqant2enfyib3KiqrPiyw== --osd-uuid 245349d5-5742-4af0-b9e3-1b7e828df542 2019-12-21T16:07:04.659 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:07:04.657+0000 7f841fb8bec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T16:07:04.661 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:07:04.659+0000 7f841fb8bec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T16:07:04.663 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:07:04.661+0000 7f841fb8bec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T16:07:04.663 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:07:04.661+0000 7f841fb8bec0 -1 bluestore(td/osd-scrub-repair/3/block) _read_bdev_label failed to open td/osd-scrub-repair/3/block: (2) No such file or directory 2019-12-21T16:07:04.663 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:07:04.662+0000 7f841fb8bec0 -1 bluestore(td/osd-scrub-repair/3/block) _read_bdev_label failed to open td/osd-scrub-repair/3/block: (2) No such file or directory 2019-12-21T16:07:04.664 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:07:04.662+0000 7f841fb8bec0 -1 bluestore(td/osd-scrub-repair/3/block) _read_bdev_label failed to open td/osd-scrub-repair/3/block: (2) No such file or directory 2019-12-21T16:07:04.727 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:07:04.725+0000 7f841fb8bec0 -1 bluestore(td/osd-scrub-repair/3) _read_fsid unparsable uuid 2019-12-21T16:07:07.730 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: local key_fn=td/osd-scrub-repair/3/keyring 2019-12-21T16:07:07.730 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: cat 2019-12-21T16:07:07.731 INFO:tasks.workunit.client.0.smithi172.stdout:adding osd3 key to auth repository 2019-12-21T16:07:07.731 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: echo adding osd3 key to auth repository 2019-12-21T16:07:07.731 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph -i td/osd-scrub-repair/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2019-12-21T16:07:08.016 INFO:tasks.workunit.client.0.smithi172.stdout:start osd.3 2019-12-21T16:07:08.017 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo start osd.3 2019-12-21T16:07:08.017 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph-osd -i 3 --fsid=973e3fc1-e63e-4bde-9db6-84c68fd5e93e --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.119575/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2019-12-21T16:07:08.017 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph osd dump --format=json 2019-12-21T16:07:08.017 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: jq '.flags_set[]' 2019-12-21T16:07:08.017 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: grep -q '"noup"' 2019-12-21T16:07:08.035 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:07:08.034+0000 7f972ca9bec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T16:07:08.037 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:07:08.036+0000 7f972ca9bec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T16:07:08.039 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:07:08.037+0000 7f972ca9bec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T16:07:08.248 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: wait_for_osd up 3 2019-12-21T16:07:08.248 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:914: wait_for_osd: local state=up 2019-12-21T16:07:08.248 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local id=3 2019-12-21T16:07:08.248 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: status=1 2019-12-21T16:07:08.248 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i=0 )) 2019-12-21T16:07:08.249 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i < 300 )) 2019-12-21T16:07:08.249 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: echo 0 2019-12-21T16:07:08.249 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: ceph osd dump 2019-12-21T16:07:08.249 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: grep 'osd.3 up' 2019-12-21T16:07:08.249 INFO:tasks.workunit.client.0.smithi172.stdout:0 2019-12-21T16:07:08.477 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: sleep 1 2019-12-21T16:07:08.555 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:07:08.553+0000 7f972ca9bec0 -1 Falling back to public interface 2019-12-21T16:07:09.317 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:07:09.315+0000 7f972ca9bec0 -1 osd.3 0 log_to_monitors {default=true} 2019-12-21T16:07:09.478 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i++ )) 2019-12-21T16:07:09.479 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i < 300 )) 2019-12-21T16:07:09.479 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: echo 1 2019-12-21T16:07:09.479 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: ceph osd dump 2019-12-21T16:07:09.479 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: grep 'osd.3 up' 2019-12-21T16:07:09.479 INFO:tasks.workunit.client.0.smithi172.stdout:1 2019-12-21T16:07:09.753 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: sleep 1 2019-12-21T16:07:10.755 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i++ )) 2019-12-21T16:07:10.755 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i < 300 )) 2019-12-21T16:07:10.755 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: echo 2 2019-12-21T16:07:10.755 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: ceph osd dump 2019-12-21T16:07:10.755 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: grep 'osd.3 up' 2019-12-21T16:07:10.756 INFO:tasks.workunit.client.0.smithi172.stdout:2 2019-12-21T16:07:10.987 INFO:tasks.workunit.client.0.smithi172.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/166885,v1:127.0.0.1:6827/166885] [v2:127.0.0.1:6828/166885,v1:127.0.0.1:6829/166885] exists,up 245349d5-5742-4af0-b9e3-1b7e828df542 2019-12-21T16:07:10.987 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=0 2019-12-21T16:07:10.988 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: break 2019-12-21T16:07:10.988 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: return 0 2019-12-21T16:07:10.988 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:724: corrupt_and_repair_jerasure: create_rbd_pool 2019-12-21T16:07:10.988 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2019-12-21T16:07:11.211 INFO:tasks.workunit.client.0.smithi172.stderr:pool 'rbd' does not exist 2019-12-21T16:07:11.219 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:534: create_rbd_pool: create_pool rbd 4 2019-12-21T16:07:11.219 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:539: create_pool: ceph osd pool create rbd 4 2019-12-21T16:07:11.630 INFO:tasks.workunit.client.0.smithi172.stderr:pool 'rbd' created 2019-12-21T16:07:11.637 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: sleep 1 2019-12-21T16:07:12.639 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: rbd pool init rbd 2019-12-21T16:07:13.626 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:725: corrupt_and_repair_jerasure: wait_for_clean 2019-12-21T16:07:13.626 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1548: wait_for_clean: local cmd= 2019-12-21T16:07:13.626 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1549: wait_for_clean: local num_active_clean=-1 2019-12-21T16:07:13.627 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1550: wait_for_clean: local cur_active_clean 2019-12-21T16:07:13.627 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1551: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2019-12-21T16:07:13.627 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1551: wait_for_clean: get_timeout_delays 90 .1 2019-12-21T16:07:13.628 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1494: get_timeout_delays: shopt -q -o xtrace 2019-12-21T16:07:13.628 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1494: get_timeout_delays: echo true 2019-12-21T16:07:13.628 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1494: get_timeout_delays: local trace=true 2019-12-21T16:07:13.628 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1495: get_timeout_delays: true 2019-12-21T16:07:13.628 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1495: get_timeout_delays: shopt -u -o xtrace 2019-12-21T16:07:13.731 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1551: wait_for_clean: local -a delays 2019-12-21T16:07:13.732 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1552: wait_for_clean: local -i loop=0 2019-12-21T16:07:13.732 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1554: wait_for_clean: flush_pg_stats 2019-12-21T16:07:13.732 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1977: flush_pg_stats: local timeout=300 2019-12-21T16:07:13.732 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1979: flush_pg_stats: ceph osd ls 2019-12-21T16:07:13.959 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1979: flush_pg_stats: ids='0 2019-12-21T16:07:13.960 INFO:tasks.workunit.client.0.smithi172.stderr:1 2019-12-21T16:07:13.960 INFO:tasks.workunit.client.0.smithi172.stderr:2 2019-12-21T16:07:13.960 INFO:tasks.workunit.client.0.smithi172.stderr:3' 2019-12-21T16:07:13.960 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1980: flush_pg_stats: seqs= 2019-12-21T16:07:13.960 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: flush_pg_stats: for osd in $ids 2019-12-21T16:07:13.961 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2019-12-21T16:07:14.078 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: seq=21474836486 2019-12-21T16:07:14.078 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: flush_pg_stats: seqs=' 0-21474836486' 2019-12-21T16:07:14.078 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: flush_pg_stats: for osd in $ids 2019-12-21T16:07:14.079 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2019-12-21T16:07:14.188 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: seq=42949672965 2019-12-21T16:07:14.188 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: flush_pg_stats: seqs=' 0-21474836486 1-42949672965' 2019-12-21T16:07:14.188 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: flush_pg_stats: for osd in $ids 2019-12-21T16:07:14.188 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2019-12-21T16:07:14.307 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: seq=64424509444 2019-12-21T16:07:14.307 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: flush_pg_stats: seqs=' 0-21474836486 1-42949672965 2-64424509444' 2019-12-21T16:07:14.308 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: flush_pg_stats: for osd in $ids 2019-12-21T16:07:14.308 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2019-12-21T16:07:14.410 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: seq=85899345923 2019-12-21T16:07:14.410 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: flush_pg_stats: seqs=' 0-21474836486 1-42949672965 2-64424509444 3-85899345923' 2019-12-21T16:07:14.410 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: flush_pg_stats: for s in $seqs 2019-12-21T16:07:14.411 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: echo 0-21474836486 2019-12-21T16:07:14.411 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: cut -d - -f 1 2019-12-21T16:07:14.411 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: osd=0 2019-12-21T16:07:14.412 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: echo 0-21474836486 2019-12-21T16:07:14.412 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: cut -d - -f 2 2019-12-21T16:07:14.413 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: seq=21474836486 2019-12-21T16:07:14.413 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1989: flush_pg_stats: echo 'waiting osd.0 seq 21474836486' 2019-12-21T16:07:14.413 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 0 2019-12-21T16:07:14.413 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.0 seq 21474836486 2019-12-21T16:07:14.640 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 21474836485 -lt 21474836486 2019-12-21T16:07:14.640 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1991: flush_pg_stats: sleep 1 2019-12-21T16:07:15.641 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1992: flush_pg_stats: '[' 300 -eq 0 ']' 2019-12-21T16:07:15.641 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 0 2019-12-21T16:07:15.881 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 21474836487 -lt 21474836486 2019-12-21T16:07:15.882 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: flush_pg_stats: for s in $seqs 2019-12-21T16:07:15.882 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: echo 1-42949672965 2019-12-21T16:07:15.882 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: cut -d - -f 1 2019-12-21T16:07:15.882 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: osd=1 2019-12-21T16:07:15.882 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: echo 1-42949672965 2019-12-21T16:07:15.882 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: cut -d - -f 2 2019-12-21T16:07:15.883 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: seq=42949672965 2019-12-21T16:07:15.883 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1989: flush_pg_stats: echo 'waiting osd.1 seq 42949672965' 2019-12-21T16:07:15.883 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 1 2019-12-21T16:07:15.883 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.1 seq 42949672965 2019-12-21T16:07:16.109 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 42949672965 -lt 42949672965 2019-12-21T16:07:16.109 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: flush_pg_stats: for s in $seqs 2019-12-21T16:07:16.109 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: echo 2-64424509444 2019-12-21T16:07:16.110 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: cut -d - -f 1 2019-12-21T16:07:16.110 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: osd=2 2019-12-21T16:07:16.111 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: echo 2-64424509444 2019-12-21T16:07:16.111 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: cut -d - -f 2 2019-12-21T16:07:16.113 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: seq=64424509444 2019-12-21T16:07:16.113 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1989: flush_pg_stats: echo 'waiting osd.2 seq 64424509444' 2019-12-21T16:07:16.113 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 2 2019-12-21T16:07:16.113 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.2 seq 64424509444 2019-12-21T16:07:16.341 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 64424509444 -lt 64424509444 2019-12-21T16:07:16.341 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: flush_pg_stats: for s in $seqs 2019-12-21T16:07:16.342 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: echo 3-85899345923 2019-12-21T16:07:16.342 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: cut -d - -f 1 2019-12-21T16:07:16.342 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: osd=3 2019-12-21T16:07:16.343 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: echo 3-85899345923 2019-12-21T16:07:16.343 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: cut -d - -f 2 2019-12-21T16:07:16.344 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: seq=85899345923 2019-12-21T16:07:16.344 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1989: flush_pg_stats: echo 'waiting osd.3 seq 85899345923' 2019-12-21T16:07:16.344 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.3 seq 85899345923 2019-12-21T16:07:16.345 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 3 2019-12-21T16:07:16.572 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 85899345924 -lt 85899345923 2019-12-21T16:07:16.574 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1555: wait_for_clean: get_num_pgs 2019-12-21T16:07:16.575 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: ceph --format json status 2019-12-21T16:07:16.575 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: jq .pgmap.num_pgs 2019-12-21T16:07:16.843 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1555: wait_for_clean: test 4 == 0 2019-12-21T16:07:16.843 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1559: wait_for_clean: true 2019-12-21T16:07:16.844 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1563: wait_for_clean: get_num_active_clean 2019-12-21T16:07:16.844 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: get_num_active_clean: local expression 2019-12-21T16:07:16.844 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2019-12-21T16:07:16.844 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("stale") | not)' 2019-12-21T16:07:16.844 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: ceph --format json pg dump pgs 2019-12-21T16:07:16.845 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2019-12-21T16:07:17.067 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1563: wait_for_clean: cur_active_clean=4 2019-12-21T16:07:17.067 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1564: wait_for_clean: get_num_pgs 2019-12-21T16:07:17.067 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: ceph --format json status 2019-12-21T16:07:17.068 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: jq .pgmap.num_pgs 2019-12-21T16:07:17.335 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1564: wait_for_clean: test 4 = 4 2019-12-21T16:07:17.335 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1564: wait_for_clean: break 2019-12-21T16:07:17.336 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: return 0 2019-12-21T16:07:17.336 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:727: corrupt_and_repair_jerasure: create_ec_pool ecpool true k=2 m=2 2019-12-21T16:07:17.336 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: create_ec_pool: local pool_name=ecpool 2019-12-21T16:07:17.336 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2179: create_ec_pool: shift 2019-12-21T16:07:17.336 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2180: create_ec_pool: local allow_overwrites=true 2019-12-21T16:07:17.337 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: create_ec_pool: shift 2019-12-21T16:07:17.337 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=2 2019-12-21T16:07:17.728 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2019-12-21T16:07:17.729 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:539: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2019-12-21T16:07:18.370 INFO:tasks.workunit.client.0.smithi172.stderr:pool 'ecpool' created 2019-12-21T16:07:18.379 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: sleep 1 2019-12-21T16:07:19.380 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: create_ec_pool: '[' true = true ']' 2019-12-21T16:07:19.380 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: create_ec_pool: ceph osd pool set ecpool allow_ec_overwrites true 2019-12-21T16:07:19.777 INFO:tasks.workunit.client.0.smithi172.stderr:set pool 2 allow_ec_overwrites to true 2019-12-21T16:07:19.786 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2191: create_ec_pool: wait_for_clean 2019-12-21T16:07:19.786 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1548: wait_for_clean: local cmd= 2019-12-21T16:07:19.786 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1549: wait_for_clean: local num_active_clean=-1 2019-12-21T16:07:19.786 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1550: wait_for_clean: local cur_active_clean 2019-12-21T16:07:19.786 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1551: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2019-12-21T16:07:19.786 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1551: wait_for_clean: get_timeout_delays 90 .1 2019-12-21T16:07:19.787 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1494: get_timeout_delays: shopt -q -o xtrace 2019-12-21T16:07:19.787 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1494: get_timeout_delays: echo true 2019-12-21T16:07:19.788 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1494: get_timeout_delays: local trace=true 2019-12-21T16:07:19.788 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1495: get_timeout_delays: true 2019-12-21T16:07:19.788 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1495: get_timeout_delays: shopt -u -o xtrace 2019-12-21T16:07:19.878 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1551: wait_for_clean: local -a delays 2019-12-21T16:07:19.879 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1552: wait_for_clean: local -i loop=0 2019-12-21T16:07:19.879 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1554: wait_for_clean: flush_pg_stats 2019-12-21T16:07:19.879 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1977: flush_pg_stats: local timeout=300 2019-12-21T16:07:19.879 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1979: flush_pg_stats: ceph osd ls 2019-12-21T16:07:20.112 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1979: flush_pg_stats: ids='0 2019-12-21T16:07:20.112 INFO:tasks.workunit.client.0.smithi172.stderr:1 2019-12-21T16:07:20.112 INFO:tasks.workunit.client.0.smithi172.stderr:2 2019-12-21T16:07:20.113 INFO:tasks.workunit.client.0.smithi172.stderr:3' 2019-12-21T16:07:20.113 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1980: flush_pg_stats: seqs= 2019-12-21T16:07:20.113 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: flush_pg_stats: for osd in $ids 2019-12-21T16:07:20.113 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2019-12-21T16:07:20.225 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: seq=21474836488 2019-12-21T16:07:20.225 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: flush_pg_stats: seqs=' 0-21474836488' 2019-12-21T16:07:20.225 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: flush_pg_stats: for osd in $ids 2019-12-21T16:07:20.226 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2019-12-21T16:07:20.345 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: seq=42949672967 2019-12-21T16:07:20.346 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: flush_pg_stats: seqs=' 0-21474836488 1-42949672967' 2019-12-21T16:07:20.346 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: flush_pg_stats: for osd in $ids 2019-12-21T16:07:20.346 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2019-12-21T16:07:20.479 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: seq=64424509446 2019-12-21T16:07:20.479 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: flush_pg_stats: seqs=' 0-21474836488 1-42949672967 2-64424509446' 2019-12-21T16:07:20.480 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: flush_pg_stats: for osd in $ids 2019-12-21T16:07:20.480 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2019-12-21T16:07:20.592 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: seq=85899345926 2019-12-21T16:07:20.592 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: flush_pg_stats: seqs=' 0-21474836488 1-42949672967 2-64424509446 3-85899345926' 2019-12-21T16:07:20.592 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: flush_pg_stats: for s in $seqs 2019-12-21T16:07:20.593 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: echo 0-21474836488 2019-12-21T16:07:20.593 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: cut -d - -f 1 2019-12-21T16:07:20.593 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: osd=0 2019-12-21T16:07:20.594 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: echo 0-21474836488 2019-12-21T16:07:20.594 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: cut -d - -f 2 2019-12-21T16:07:20.595 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: seq=21474836488 2019-12-21T16:07:20.595 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1989: flush_pg_stats: echo 'waiting osd.0 seq 21474836488' 2019-12-21T16:07:20.595 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 0 2019-12-21T16:07:20.595 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.0 seq 21474836488 2019-12-21T16:07:20.820 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 21474836487 -lt 21474836488 2019-12-21T16:07:20.821 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1991: flush_pg_stats: sleep 1 2019-12-21T16:07:21.822 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1992: flush_pg_stats: '[' 300 -eq 0 ']' 2019-12-21T16:07:21.822 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 0 2019-12-21T16:07:22.051 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 21474836489 -lt 21474836488 2019-12-21T16:07:22.051 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: flush_pg_stats: for s in $seqs 2019-12-21T16:07:22.051 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: echo 1-42949672967 2019-12-21T16:07:22.052 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: cut -d - -f 1 2019-12-21T16:07:22.052 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: osd=1 2019-12-21T16:07:22.052 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: echo 1-42949672967 2019-12-21T16:07:22.053 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: cut -d - -f 2 2019-12-21T16:07:22.054 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: seq=42949672967 2019-12-21T16:07:22.054 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1989: flush_pg_stats: echo 'waiting osd.1 seq 42949672967' 2019-12-21T16:07:22.054 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 1 2019-12-21T16:07:22.054 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.1 seq 42949672967 2019-12-21T16:07:22.281 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 42949672967 -lt 42949672967 2019-12-21T16:07:22.281 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: flush_pg_stats: for s in $seqs 2019-12-21T16:07:22.281 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: echo 2-64424509446 2019-12-21T16:07:22.281 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: cut -d - -f 1 2019-12-21T16:07:22.282 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: osd=2 2019-12-21T16:07:22.282 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: echo 2-64424509446 2019-12-21T16:07:22.283 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: cut -d - -f 2 2019-12-21T16:07:22.283 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: seq=64424509446 2019-12-21T16:07:22.284 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1989: flush_pg_stats: echo 'waiting osd.2 seq 64424509446' 2019-12-21T16:07:22.284 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 2 2019-12-21T16:07:22.284 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.2 seq 64424509446 2019-12-21T16:07:22.512 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 64424509446 -lt 64424509446 2019-12-21T16:07:22.512 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: flush_pg_stats: for s in $seqs 2019-12-21T16:07:22.512 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: echo 3-85899345926 2019-12-21T16:07:22.513 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: cut -d - -f 1 2019-12-21T16:07:22.513 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: osd=3 2019-12-21T16:07:22.514 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: echo 3-85899345926 2019-12-21T16:07:22.514 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: cut -d - -f 2 2019-12-21T16:07:22.515 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.3 seq 85899345926 2019-12-21T16:07:22.516 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: seq=85899345926 2019-12-21T16:07:22.516 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1989: flush_pg_stats: echo 'waiting osd.3 seq 85899345926' 2019-12-21T16:07:22.516 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 3 2019-12-21T16:07:22.746 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 85899345926 -lt 85899345926 2019-12-21T16:07:22.747 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1555: wait_for_clean: get_num_pgs 2019-12-21T16:07:22.747 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: ceph --format json status 2019-12-21T16:07:22.747 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: jq .pgmap.num_pgs 2019-12-21T16:07:23.017 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1555: wait_for_clean: test 5 == 0 2019-12-21T16:07:23.017 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1559: wait_for_clean: true 2019-12-21T16:07:23.017 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1563: wait_for_clean: get_num_active_clean 2019-12-21T16:07:23.017 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: get_num_active_clean: local expression 2019-12-21T16:07:23.018 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2019-12-21T16:07:23.018 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("stale") | not)' 2019-12-21T16:07:23.018 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: ceph --format json pg dump pgs 2019-12-21T16:07:23.018 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2019-12-21T16:07:23.240 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1563: wait_for_clean: cur_active_clean=5 2019-12-21T16:07:23.241 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1564: wait_for_clean: get_num_pgs 2019-12-21T16:07:23.241 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: ceph --format json status 2019-12-21T16:07:23.241 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: jq .pgmap.num_pgs 2019-12-21T16:07:23.511 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1564: wait_for_clean: test 5 = 5 2019-12-21T16:07:23.511 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1564: wait_for_clean: break 2019-12-21T16:07:23.512 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: return 0 2019-12-21T16:07:23.512 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2192: create_ec_pool: return 0 2019-12-21T16:07:23.512 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:728: corrupt_and_repair_jerasure: corrupt_and_repair_erasure_coded td/osd-scrub-repair ecpool 2019-12-21T16:07:23.512 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: local dir=td/osd-scrub-repair 2019-12-21T16:07:23.512 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:255: corrupt_and_repair_erasure_coded: local poolname=ecpool 2019-12-21T16:07:23.512 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:257: corrupt_and_repair_erasure_coded: add_something td/osd-scrub-repair ecpool 2019-12-21T16:07:23.512 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:68: add_something: local dir=td/osd-scrub-repair 2019-12-21T16:07:23.513 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:69: add_something: local poolname=ecpool 2019-12-21T16:07:23.513 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:70: add_something: local obj=SOMETHING 2019-12-21T16:07:23.513 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local scrub=noscrub 2019-12-21T16:07:23.513 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: '[' noscrub = noscrub ']' 2019-12-21T16:07:23.513 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:75: add_something: ceph osd set noscrub 2019-12-21T16:07:23.862 INFO:tasks.workunit.client.0.smithi172.stderr:noscrub is set 2019-12-21T16:07:23.869 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: ceph osd set nodeep-scrub 2019-12-21T16:07:24.304 INFO:tasks.workunit.client.0.smithi172.stderr:nodeep-scrub is set 2019-12-21T16:07:24.312 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:82: add_something: local payload=ABCDEF 2019-12-21T16:07:24.312 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:83: add_something: echo ABCDEF 2019-12-21T16:07:24.312 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:84: add_something: rados --pool ecpool put SOMETHING td/osd-scrub-repair/ORIGINAL 2019-12-21T16:07:24.488 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:259: corrupt_and_repair_erasure_coded: get_primary ecpool SOMETHING 2019-12-21T16:07:24.488 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_primary: local poolname=ecpool 2019-12-21T16:07:24.488 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1133: get_primary: local objectname=SOMETHING 2019-12-21T16:07:24.488 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1135: get_primary: ceph --format json osd map ecpool SOMETHING 2019-12-21T16:07:24.489 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1136: get_primary: jq .acting_primary 2019-12-21T16:07:24.719 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:259: corrupt_and_repair_erasure_coded: local primary=3 2019-12-21T16:07:24.720 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:260: corrupt_and_repair_erasure_coded: osds=($(get_osds $poolname SOMETHING | sed -e "s/$primary//")) 2019-12-21T16:07:24.720 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:260: corrupt_and_repair_erasure_coded: get_osds ecpool SOMETHING 2019-12-21T16:07:24.721 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:957: get_osds: local poolname=ecpool 2019-12-21T16:07:24.721 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:260: corrupt_and_repair_erasure_coded: sed -e s/3// 2019-12-21T16:07:24.721 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:958: get_osds: local objectname=SOMETHING 2019-12-21T16:07:24.721 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: get_osds: ceph --format json osd map ecpool SOMETHING 2019-12-21T16:07:24.721 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: get_osds: jq '.acting | .[]' 2019-12-21T16:07:24.951 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: get_osds: local 'osds=3 2019-12-21T16:07:24.951 INFO:tasks.workunit.client.0.smithi172.stderr:1 2019-12-21T16:07:24.951 INFO:tasks.workunit.client.0.smithi172.stderr:2 2019-12-21T16:07:24.951 INFO:tasks.workunit.client.0.smithi172.stderr:0' 2019-12-21T16:07:24.951 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: echo 3 1 2 0 2019-12-21T16:07:24.953 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:260: corrupt_and_repair_erasure_coded: local -a osds 2019-12-21T16:07:24.954 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:261: corrupt_and_repair_erasure_coded: local not_primary_first=1 2019-12-21T16:07:24.954 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:262: corrupt_and_repair_erasure_coded: local not_primary_second=2 2019-12-21T16:07:24.954 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:265: corrupt_and_repair_erasure_coded: corrupt_and_repair_one td/osd-scrub-repair ecpool 3 2019-12-21T16:07:24.954 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:232: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2019-12-21T16:07:24.954 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:233: corrupt_and_repair_one: local poolname=ecpool 2019-12-21T16:07:24.954 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:234: corrupt_and_repair_one: local osd=3 2019-12-21T16:07:24.954 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:239: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2019-12-21T16:07:24.955 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: objectstore_tool: local dir=td/osd-scrub-repair 2019-12-21T16:07:24.955 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: objectstore_tool: shift 2019-12-21T16:07:24.955 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1235: objectstore_tool: local id=3 2019-12-21T16:07:24.955 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1236: objectstore_tool: shift 2019-12-21T16:07:24.955 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1238: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING remove 2019-12-21T16:07:24.955 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1205: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2019-12-21T16:07:24.956 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nowait: shift 2019-12-21T16:07:24.956 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1207: _objectstore_tool_nowait: local id=3 2019-12-21T16:07:24.956 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1208: _objectstore_tool_nowait: shift 2019-12-21T16:07:24.956 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1210: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2019-12-21T16:07:24.956 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2019-12-21T16:07:24.956 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2019-12-21T16:07:24.956 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2019-12-21T16:07:24.957 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2019-12-21T16:07:24.957 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2019-12-21T16:07:25.058 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2019-12-21T16:07:25.058 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1212: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING remove 2019-12-21T16:07:25.058 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1193: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2019-12-21T16:07:25.058 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1194: _objectstore_tool_nodown: shift 2019-12-21T16:07:25.058 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1195: _objectstore_tool_nodown: local id=3 2019-12-21T16:07:25.058 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: _objectstore_tool_nodown: shift 2019-12-21T16:07:25.059 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2019-12-21T16:07:25.059 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING remove 2019-12-21T16:07:25.988 INFO:tasks.workunit.client.0.smithi172.stdout:remove 0#2:eb822e21:::SOMETHING:head# 2019-12-21T16:07:26.507 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1213: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2019-12-21T16:07:26.507 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:842: activate_osd: local dir=td/osd-scrub-repair 2019-12-21T16:07:26.507 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: shift 2019-12-21T16:07:26.508 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: local id=3 2019-12-21T16:07:26.508 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: shift 2019-12-21T16:07:26.508 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local osd_data=td/osd-scrub-repair/3 2019-12-21T16:07:26.508 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local 'ceph_args=--fsid=973e3fc1-e63e-4bde-9db6-84c68fd5e93e --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2019-12-21T16:07:26.508 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2019-12-21T16:07:26.508 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-journal-size=100' 2019-12-21T16:07:26.509 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2019-12-21T16:07:26.509 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2019-12-21T16:07:26.509 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2019-12-21T16:07:26.509 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --chdir=' 2019-12-21T16:07:26.509 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+= 2019-12-21T16:07:26.509 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2019-12-21T16:07:26.509 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: get_asok_path 2019-12-21T16:07:26.509 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2019-12-21T16:07:26.510 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2019-12-21T16:07:26.510 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2019-12-21T16:07:26.510 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T16:07:26.510 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.119575 2019-12-21T16:07:26.510 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.119575/$cluster-$name.asok' 2019-12-21T16:07:26.510 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.119575/$cluster-$name.asok' 2019-12-21T16:07:26.510 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --debug-osd=20' 2019-12-21T16:07:26.511 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2019-12-21T16:07:26.511 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2019-12-21T16:07:26.511 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2019-12-21T16:07:26.511 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2019-12-21T16:07:26.511 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2019-12-21T16:07:26.511 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' ' 2019-12-21T16:07:26.511 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+= 2019-12-21T16:07:26.511 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: mkdir -p td/osd-scrub-repair/3 2019-12-21T16:07:26.517 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: echo start osd.3 2019-12-21T16:07:26.518 INFO:tasks.workunit.client.0.smithi172.stderr:start osd.3 2019-12-21T16:07:26.532 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph-osd -i 3 --fsid=973e3fc1-e63e-4bde-9db6-84c68fd5e93e --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.119575/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2019-12-21T16:07:26.532 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: cat td/osd-scrub-repair/3/whoami 2019-12-21T16:07:26.533 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: '[' 3 = 3 ']' 2019-12-21T16:07:26.533 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph osd dump --format=json 2019-12-21T16:07:26.533 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: jq '.flags_set[]' 2019-12-21T16:07:26.533 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: grep -q '"noup"' 2019-12-21T16:07:26.533 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:07:26.527+0000 7f7b05e2dec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T16:07:26.534 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:07:26.529+0000 7f7b05e2dec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T16:07:26.534 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:07:26.531+0000 7f7b05e2dec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T16:07:26.749 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: wait_for_osd up 3 2019-12-21T16:07:26.749 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:914: wait_for_osd: local state=up 2019-12-21T16:07:26.749 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local id=3 2019-12-21T16:07:26.749 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: status=1 2019-12-21T16:07:26.749 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i=0 )) 2019-12-21T16:07:26.749 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i < 300 )) 2019-12-21T16:07:26.750 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: echo 0 2019-12-21T16:07:26.750 INFO:tasks.workunit.client.0.smithi172.stderr:0 2019-12-21T16:07:26.750 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: ceph osd dump 2019-12-21T16:07:26.750 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: grep 'osd.3 up' 2019-12-21T16:07:26.975 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: sleep 1 2019-12-21T16:07:27.050 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:07:27.047+0000 7f7b05e2dec0 -1 Falling back to public interface 2019-12-21T16:07:27.901 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:07:27.898+0000 7f7b05e2dec0 -1 osd.3 37 log_to_monitors {default=true} 2019-12-21T16:07:27.977 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i++ )) 2019-12-21T16:07:27.977 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i < 300 )) 2019-12-21T16:07:27.977 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: echo 1 2019-12-21T16:07:27.977 INFO:tasks.workunit.client.0.smithi172.stderr:1 2019-12-21T16:07:27.978 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: ceph osd dump 2019-12-21T16:07:27.978 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: grep 'osd.3 up' 2019-12-21T16:07:28.296 INFO:tasks.workunit.client.0.smithi172.stderr:osd.3 up in weight 1 up_from 41 up_thru 32 down_at 38 last_clean_interval [20,37) [v2:127.0.0.1:6826/168368,v1:127.0.0.1:6827/168368] [v2:127.0.0.1:6828/168368,v1:127.0.0.1:6829/168368] exists,up 245349d5-5742-4af0-b9e3-1b7e828df542 2019-12-21T16:07:28.296 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=0 2019-12-21T16:07:28.296 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: break 2019-12-21T16:07:28.297 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: return 0 2019-12-21T16:07:28.297 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1239: objectstore_tool: wait_for_clean 2019-12-21T16:07:28.297 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1548: wait_for_clean: local cmd= 2019-12-21T16:07:28.297 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1549: wait_for_clean: local num_active_clean=-1 2019-12-21T16:07:28.297 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1550: wait_for_clean: local cur_active_clean 2019-12-21T16:07:28.297 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1551: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2019-12-21T16:07:28.298 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1551: wait_for_clean: get_timeout_delays 90 .1 2019-12-21T16:07:28.298 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1494: get_timeout_delays: shopt -q -o xtrace 2019-12-21T16:07:28.298 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1494: get_timeout_delays: echo true 2019-12-21T16:07:28.298 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1494: get_timeout_delays: local trace=true 2019-12-21T16:07:28.298 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1495: get_timeout_delays: true 2019-12-21T16:07:28.298 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1495: get_timeout_delays: shopt -u -o xtrace 2019-12-21T16:07:28.389 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1551: wait_for_clean: local -a delays 2019-12-21T16:07:28.389 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1552: wait_for_clean: local -i loop=0 2019-12-21T16:07:28.389 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1554: wait_for_clean: flush_pg_stats 2019-12-21T16:07:28.390 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1977: flush_pg_stats: local timeout=300 2019-12-21T16:07:28.390 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1979: flush_pg_stats: ceph osd ls 2019-12-21T16:07:28.645 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1979: flush_pg_stats: ids='0 2019-12-21T16:07:28.645 INFO:tasks.workunit.client.0.smithi172.stderr:1 2019-12-21T16:07:28.645 INFO:tasks.workunit.client.0.smithi172.stderr:2 2019-12-21T16:07:28.646 INFO:tasks.workunit.client.0.smithi172.stderr:3' 2019-12-21T16:07:28.646 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1980: flush_pg_stats: seqs= 2019-12-21T16:07:28.646 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: flush_pg_stats: for osd in $ids 2019-12-21T16:07:28.646 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2019-12-21T16:07:28.763 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: seq=21474836491 2019-12-21T16:07:28.763 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: flush_pg_stats: seqs=' 0-21474836491' 2019-12-21T16:07:28.764 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: flush_pg_stats: for osd in $ids 2019-12-21T16:07:28.764 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2019-12-21T16:07:28.873 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: seq=42949672970 2019-12-21T16:07:28.874 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: flush_pg_stats: seqs=' 0-21474836491 1-42949672970' 2019-12-21T16:07:28.874 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: flush_pg_stats: for osd in $ids 2019-12-21T16:07:28.874 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2019-12-21T16:07:28.983 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: seq=64424509449 2019-12-21T16:07:28.984 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: flush_pg_stats: seqs=' 0-21474836491 1-42949672970 2-64424509449' 2019-12-21T16:07:28.984 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: flush_pg_stats: for osd in $ids 2019-12-21T16:07:28.984 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2019-12-21T16:07:29.094 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: seq=176093659138 2019-12-21T16:07:29.094 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: flush_pg_stats: seqs=' 0-21474836491 1-42949672970 2-64424509449 3-176093659138' 2019-12-21T16:07:29.094 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: flush_pg_stats: for s in $seqs 2019-12-21T16:07:29.095 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: echo 0-21474836491 2019-12-21T16:07:29.095 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: cut -d - -f 1 2019-12-21T16:07:29.096 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: osd=0 2019-12-21T16:07:29.096 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: echo 0-21474836491 2019-12-21T16:07:29.096 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: cut -d - -f 2 2019-12-21T16:07:29.097 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: seq=21474836491 2019-12-21T16:07:29.097 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1989: flush_pg_stats: echo 'waiting osd.0 seq 21474836491' 2019-12-21T16:07:29.097 INFO:tasks.workunit.client.0.smithi172.stderr:waiting osd.0 seq 21474836491 2019-12-21T16:07:29.098 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 0 2019-12-21T16:07:29.327 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 21474836490 -lt 21474836491 2019-12-21T16:07:29.327 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1991: flush_pg_stats: sleep 1 2019-12-21T16:07:30.328 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1992: flush_pg_stats: '[' 300 -eq 0 ']' 2019-12-21T16:07:30.328 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 0 2019-12-21T16:07:30.565 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 21474836491 -lt 21474836491 2019-12-21T16:07:30.566 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: flush_pg_stats: for s in $seqs 2019-12-21T16:07:30.566 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: echo 1-42949672970 2019-12-21T16:07:30.566 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: cut -d - -f 1 2019-12-21T16:07:30.567 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: osd=1 2019-12-21T16:07:30.567 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: echo 1-42949672970 2019-12-21T16:07:30.567 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: cut -d - -f 2 2019-12-21T16:07:30.568 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: seq=42949672970 2019-12-21T16:07:30.568 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1989: flush_pg_stats: echo 'waiting osd.1 seq 42949672970' 2019-12-21T16:07:30.569 INFO:tasks.workunit.client.0.smithi172.stderr:waiting osd.1 seq 42949672970 2019-12-21T16:07:30.569 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 1 2019-12-21T16:07:30.796 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 42949672970 -lt 42949672970 2019-12-21T16:07:30.797 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: flush_pg_stats: for s in $seqs 2019-12-21T16:07:30.797 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: echo 2-64424509449 2019-12-21T16:07:30.797 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: cut -d - -f 1 2019-12-21T16:07:30.798 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: osd=2 2019-12-21T16:07:30.798 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: echo 2-64424509449 2019-12-21T16:07:30.799 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: cut -d - -f 2 2019-12-21T16:07:30.802 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: seq=64424509449 2019-12-21T16:07:30.802 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1989: flush_pg_stats: echo 'waiting osd.2 seq 64424509449' 2019-12-21T16:07:30.803 INFO:tasks.workunit.client.0.smithi172.stderr:waiting osd.2 seq 64424509449 2019-12-21T16:07:30.803 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 2 2019-12-21T16:07:31.027 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 64424509449 -lt 64424509449 2019-12-21T16:07:31.027 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: flush_pg_stats: for s in $seqs 2019-12-21T16:07:31.028 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: echo 3-176093659138 2019-12-21T16:07:31.028 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: cut -d - -f 1 2019-12-21T16:07:31.029 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: osd=3 2019-12-21T16:07:31.029 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: echo 3-176093659138 2019-12-21T16:07:31.030 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: cut -d - -f 2 2019-12-21T16:07:31.030 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: seq=176093659138 2019-12-21T16:07:31.030 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1989: flush_pg_stats: echo 'waiting osd.3 seq 176093659138' 2019-12-21T16:07:31.030 INFO:tasks.workunit.client.0.smithi172.stderr:waiting osd.3 seq 176093659138 2019-12-21T16:07:31.031 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 3 2019-12-21T16:07:31.258 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 176093659138 -lt 176093659138 2019-12-21T16:07:31.258 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1555: wait_for_clean: get_num_pgs 2019-12-21T16:07:31.258 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: ceph --format json status 2019-12-21T16:07:31.259 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: jq .pgmap.num_pgs 2019-12-21T16:07:31.529 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1555: wait_for_clean: test 5 == 0 2019-12-21T16:07:31.530 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1559: wait_for_clean: true 2019-12-21T16:07:31.530 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1563: wait_for_clean: get_num_active_clean 2019-12-21T16:07:31.530 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: get_num_active_clean: local expression 2019-12-21T16:07:31.530 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2019-12-21T16:07:31.530 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("stale") | not)' 2019-12-21T16:07:31.530 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: ceph --format json pg dump pgs 2019-12-21T16:07:31.530 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2019-12-21T16:07:31.752 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1563: wait_for_clean: cur_active_clean=5 2019-12-21T16:07:31.753 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1564: wait_for_clean: get_num_pgs 2019-12-21T16:07:31.753 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: ceph --format json status 2019-12-21T16:07:31.753 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: jq .pgmap.num_pgs 2019-12-21T16:07:32.021 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1564: wait_for_clean: test 5 = 5 2019-12-21T16:07:32.021 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1564: wait_for_clean: break 2019-12-21T16:07:32.022 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: return 0 2019-12-21T16:07:32.022 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:243: corrupt_and_repair_one: get_pg ecpool SOMETHING 2019-12-21T16:07:32.022 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1029: get_pg: local poolname=ecpool 2019-12-21T16:07:32.022 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1030: get_pg: local objectname=SOMETHING 2019-12-21T16:07:32.023 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1032: get_pg: ceph --format json osd map ecpool SOMETHING 2019-12-21T16:07:32.023 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1032: get_pg: jq -r .pgid 2019-12-21T16:07:32.253 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:243: corrupt_and_repair_one: local pg=2.0 2019-12-21T16:07:32.254 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:244: corrupt_and_repair_one: repair 2.0 2019-12-21T16:07:32.254 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: repair: local pgid=2.0 2019-12-21T16:07:32.254 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1665: repair: get_last_scrub_stamp 2.0 2019-12-21T16:07:32.254 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=2.0 2019-12-21T16:07:32.254 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T16:07:32.255 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T16:07:32.255 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2019-12-21T16:07:32.477 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1665: repair: local last_scrub=2019-12-21T16:07:18.258863+0000 2019-12-21T16:07:32.477 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1666: repair: ceph pg repair 2.0 2019-12-21T16:07:32.693 INFO:tasks.workunit.client.0.smithi172.stderr:instructing pg 2.0s0 on osd.3 to repair 2019-12-21T16:07:32.702 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: repair: wait_for_scrub 2.0 2019-12-21T16:07:18.258863+0000 2019-12-21T16:07:32.702 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1789: wait_for_scrub: local pgid=2.0 2019-12-21T16:07:32.702 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1790: wait_for_scrub: local last_scrub=2019-12-21T16:07:18.258863+0000 2019-12-21T16:07:32.702 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1791: wait_for_scrub: local sname=last_scrub_stamp 2019-12-21T16:07:32.702 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i=0 )) 2019-12-21T16:07:32.703 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T16:07:32.703 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2019-12-21T16:07:32.703 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=2.0 2019-12-21T16:07:32.703 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T16:07:32.703 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T16:07:32.703 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2019-12-21T16:07:32.924 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T16:07:18.258863+0000 '>' 2019-12-21T16:07:18.258863+0000 2019-12-21T16:07:32.924 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T16:07:33.925 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T16:07:33.925 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T16:07:33.926 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2019-12-21T16:07:33.926 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=2.0 2019-12-21T16:07:33.927 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T16:07:33.927 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T16:07:33.927 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2019-12-21T16:07:34.157 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T16:07:18.258863+0000 '>' 2019-12-21T16:07:18.258863+0000 2019-12-21T16:07:34.158 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T16:07:35.159 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T16:07:35.159 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T16:07:35.160 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2019-12-21T16:07:35.160 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=2.0 2019-12-21T16:07:35.160 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T16:07:35.160 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T16:07:35.160 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2019-12-21T16:07:35.394 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T16:07:18.258863+0000 '>' 2019-12-21T16:07:18.258863+0000 2019-12-21T16:07:35.394 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T16:07:36.395 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T16:07:36.395 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T16:07:36.396 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2019-12-21T16:07:36.396 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=2.0 2019-12-21T16:07:36.397 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T16:07:36.397 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T16:07:36.397 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2019-12-21T16:07:36.629 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T16:07:18.258863+0000 '>' 2019-12-21T16:07:18.258863+0000 2019-12-21T16:07:36.629 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T16:07:37.630 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T16:07:37.630 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T16:07:37.631 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2019-12-21T16:07:37.631 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=2.0 2019-12-21T16:07:37.631 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T16:07:37.631 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T16:07:37.632 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2019-12-21T16:07:37.863 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T16:07:18.258863+0000 '>' 2019-12-21T16:07:18.258863+0000 2019-12-21T16:07:37.863 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T16:07:38.864 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T16:07:38.865 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T16:07:38.865 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2019-12-21T16:07:38.866 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=2.0 2019-12-21T16:07:38.866 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T16:07:38.866 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T16:07:38.866 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2019-12-21T16:07:39.098 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T16:07:18.258863+0000 '>' 2019-12-21T16:07:18.258863+0000 2019-12-21T16:07:39.098 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T16:07:40.099 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T16:07:40.099 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T16:07:40.100 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2019-12-21T16:07:40.100 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=2.0 2019-12-21T16:07:40.100 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T16:07:40.100 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T16:07:40.100 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2019-12-21T16:07:40.333 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T16:07:34.952470+0000 '>' 2019-12-21T16:07:18.258863+0000 2019-12-21T16:07:40.333 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1795: wait_for_scrub: return 0 2019-12-21T16:07:40.333 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:248: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2019-12-21T16:07:40.333 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: objectstore_tool: local dir=td/osd-scrub-repair 2019-12-21T16:07:40.334 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: objectstore_tool: shift 2019-12-21T16:07:40.334 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1235: objectstore_tool: local id=3 2019-12-21T16:07:40.334 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1236: objectstore_tool: shift 2019-12-21T16:07:40.334 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1238: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING list-attrs 2019-12-21T16:07:40.334 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1205: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2019-12-21T16:07:40.334 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nowait: shift 2019-12-21T16:07:40.334 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1207: _objectstore_tool_nowait: local id=3 2019-12-21T16:07:40.335 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1208: _objectstore_tool_nowait: shift 2019-12-21T16:07:40.335 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1210: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2019-12-21T16:07:40.335 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2019-12-21T16:07:40.335 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2019-12-21T16:07:40.335 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2019-12-21T16:07:40.335 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2019-12-21T16:07:40.335 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2019-12-21T16:07:40.437 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2019-12-21T16:07:40.438 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1212: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING list-attrs 2019-12-21T16:07:40.438 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1193: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2019-12-21T16:07:40.438 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1194: _objectstore_tool_nodown: shift 2019-12-21T16:07:40.438 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1195: _objectstore_tool_nodown: local id=3 2019-12-21T16:07:40.438 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: _objectstore_tool_nodown: shift 2019-12-21T16:07:40.438 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2019-12-21T16:07:40.439 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING list-attrs 2019-12-21T16:07:41.102 INFO:tasks.workunit.client.0.smithi172.stderr:Error getting attr on : 2.0s0_head,0#-4:00000000:::scrub_2.0s0:head#, (61) No data available 2019-12-21T16:07:41.102 INFO:tasks.workunit.client.0.smithi172.stdout:_ 2019-12-21T16:07:41.102 INFO:tasks.workunit.client.0.smithi172.stdout:hinfo_key 2019-12-21T16:07:41.102 INFO:tasks.workunit.client.0.smithi172.stdout:snapset 2019-12-21T16:07:41.620 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1213: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2019-12-21T16:07:41.621 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:842: activate_osd: local dir=td/osd-scrub-repair 2019-12-21T16:07:41.621 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: shift 2019-12-21T16:07:41.621 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: local id=3 2019-12-21T16:07:41.621 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: shift 2019-12-21T16:07:41.621 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local osd_data=td/osd-scrub-repair/3 2019-12-21T16:07:41.621 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local 'ceph_args=--fsid=973e3fc1-e63e-4bde-9db6-84c68fd5e93e --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2019-12-21T16:07:41.621 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2019-12-21T16:07:41.622 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-journal-size=100' 2019-12-21T16:07:41.622 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2019-12-21T16:07:41.622 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2019-12-21T16:07:41.622 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2019-12-21T16:07:41.622 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --chdir=' 2019-12-21T16:07:41.622 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+= 2019-12-21T16:07:41.622 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2019-12-21T16:07:41.623 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: get_asok_path 2019-12-21T16:07:41.623 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2019-12-21T16:07:41.623 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2019-12-21T16:07:41.623 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2019-12-21T16:07:41.623 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T16:07:41.623 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.119575 2019-12-21T16:07:41.623 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.119575/$cluster-$name.asok' 2019-12-21T16:07:41.624 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.119575/$cluster-$name.asok' 2019-12-21T16:07:41.624 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --debug-osd=20' 2019-12-21T16:07:41.624 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2019-12-21T16:07:41.624 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2019-12-21T16:07:41.624 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2019-12-21T16:07:41.624 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2019-12-21T16:07:41.624 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2019-12-21T16:07:41.625 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' ' 2019-12-21T16:07:41.625 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+= 2019-12-21T16:07:41.625 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: mkdir -p td/osd-scrub-repair/3 2019-12-21T16:07:41.626 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: echo start osd.3 2019-12-21T16:07:41.626 INFO:tasks.workunit.client.0.smithi172.stderr:start osd.3 2019-12-21T16:07:41.626 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph-osd -i 3 --fsid=973e3fc1-e63e-4bde-9db6-84c68fd5e93e --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.119575/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2019-12-21T16:07:41.626 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: cat td/osd-scrub-repair/3/whoami 2019-12-21T16:07:41.626 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: '[' 3 = 3 ']' 2019-12-21T16:07:41.627 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph osd dump --format=json 2019-12-21T16:07:41.627 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: jq '.flags_set[]' 2019-12-21T16:07:41.627 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: grep -q '"noup"' 2019-12-21T16:07:41.643 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:07:41.641+0000 7f2d913cdec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T16:07:41.645 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:07:41.643+0000 7f2d913cdec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T16:07:41.647 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:07:41.646+0000 7f2d913cdec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T16:07:41.859 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: wait_for_osd up 3 2019-12-21T16:07:41.859 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:914: wait_for_osd: local state=up 2019-12-21T16:07:41.859 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local id=3 2019-12-21T16:07:41.860 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: status=1 2019-12-21T16:07:41.860 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i=0 )) 2019-12-21T16:07:41.860 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i < 300 )) 2019-12-21T16:07:41.860 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: echo 0 2019-12-21T16:07:41.860 INFO:tasks.workunit.client.0.smithi172.stderr:0 2019-12-21T16:07:41.860 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: ceph osd dump 2019-12-21T16:07:41.860 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: grep 'osd.3 up' 2019-12-21T16:07:42.085 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: sleep 1 2019-12-21T16:07:42.167 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:07:42.166+0000 7f2d913cdec0 -1 Falling back to public interface 2019-12-21T16:07:42.871 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:07:42.869+0000 7f2d913cdec0 -1 osd.3 43 log_to_monitors {default=true} 2019-12-21T16:07:43.087 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i++ )) 2019-12-21T16:07:43.087 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i < 300 )) 2019-12-21T16:07:43.087 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: echo 1 2019-12-21T16:07:43.087 INFO:tasks.workunit.client.0.smithi172.stderr:1 2019-12-21T16:07:43.087 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: ceph osd dump 2019-12-21T16:07:43.088 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: grep 'osd.3 up' 2019-12-21T16:07:43.324 INFO:tasks.workunit.client.0.smithi172.stderr:osd.3 up in weight 1 up_from 46 up_thru 46 down_at 44 last_clean_interval [41,43) [v2:127.0.0.1:6826/169359,v1:127.0.0.1:6827/169359] [v2:127.0.0.1:6828/169359,v1:127.0.0.1:6829/169359] exists,up 245349d5-5742-4af0-b9e3-1b7e828df542 2019-12-21T16:07:43.325 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=0 2019-12-21T16:07:43.325 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: break 2019-12-21T16:07:43.325 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: return 0 2019-12-21T16:07:43.325 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1239: objectstore_tool: wait_for_clean 2019-12-21T16:07:43.325 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1548: wait_for_clean: local cmd= 2019-12-21T16:07:43.325 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1549: wait_for_clean: local num_active_clean=-1 2019-12-21T16:07:43.325 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1550: wait_for_clean: local cur_active_clean 2019-12-21T16:07:43.326 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1551: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2019-12-21T16:07:43.326 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1551: wait_for_clean: get_timeout_delays 90 .1 2019-12-21T16:07:43.326 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1494: get_timeout_delays: shopt -q -o xtrace 2019-12-21T16:07:43.326 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1494: get_timeout_delays: echo true 2019-12-21T16:07:43.326 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1494: get_timeout_delays: local trace=true 2019-12-21T16:07:43.326 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1495: get_timeout_delays: true 2019-12-21T16:07:43.326 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1495: get_timeout_delays: shopt -u -o xtrace 2019-12-21T16:07:43.416 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1551: wait_for_clean: local -a delays 2019-12-21T16:07:43.417 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1552: wait_for_clean: local -i loop=0 2019-12-21T16:07:43.417 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1554: wait_for_clean: flush_pg_stats 2019-12-21T16:07:43.417 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1977: flush_pg_stats: local timeout=300 2019-12-21T16:07:43.417 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1979: flush_pg_stats: ceph osd ls 2019-12-21T16:07:43.644 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1979: flush_pg_stats: ids='0 2019-12-21T16:07:43.644 INFO:tasks.workunit.client.0.smithi172.stderr:1 2019-12-21T16:07:43.644 INFO:tasks.workunit.client.0.smithi172.stderr:2 2019-12-21T16:07:43.645 INFO:tasks.workunit.client.0.smithi172.stderr:3' 2019-12-21T16:07:43.645 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1980: flush_pg_stats: seqs= 2019-12-21T16:07:43.645 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: flush_pg_stats: for osd in $ids 2019-12-21T16:07:43.645 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2019-12-21T16:07:43.785 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: seq=21474836495 2019-12-21T16:07:43.785 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: flush_pg_stats: seqs=' 0-21474836495' 2019-12-21T16:07:43.786 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: flush_pg_stats: for osd in $ids 2019-12-21T16:07:43.786 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2019-12-21T16:07:43.903 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: seq=42949672974 2019-12-21T16:07:43.903 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: flush_pg_stats: seqs=' 0-21474836495 1-42949672974' 2019-12-21T16:07:43.903 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: flush_pg_stats: for osd in $ids 2019-12-21T16:07:43.903 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2019-12-21T16:07:44.013 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: seq=64424509453 2019-12-21T16:07:44.013 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: flush_pg_stats: seqs=' 0-21474836495 1-42949672974 2-64424509453' 2019-12-21T16:07:44.013 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: flush_pg_stats: for osd in $ids 2019-12-21T16:07:44.014 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2019-12-21T16:07:44.125 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: seq=197568495618 2019-12-21T16:07:44.125 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: flush_pg_stats: seqs=' 0-21474836495 1-42949672974 2-64424509453 3-197568495618' 2019-12-21T16:07:44.125 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: flush_pg_stats: for s in $seqs 2019-12-21T16:07:44.125 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: echo 0-21474836495 2019-12-21T16:07:44.126 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: cut -d - -f 1 2019-12-21T16:07:44.126 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: osd=0 2019-12-21T16:07:44.127 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: echo 0-21474836495 2019-12-21T16:07:44.127 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: cut -d - -f 2 2019-12-21T16:07:44.128 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: seq=21474836495 2019-12-21T16:07:44.128 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1989: flush_pg_stats: echo 'waiting osd.0 seq 21474836495' 2019-12-21T16:07:44.128 INFO:tasks.workunit.client.0.smithi172.stderr:waiting osd.0 seq 21474836495 2019-12-21T16:07:44.128 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 0 2019-12-21T16:07:44.357 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 21474836494 -lt 21474836495 2019-12-21T16:07:44.357 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1991: flush_pg_stats: sleep 1 2019-12-21T16:07:45.358 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1992: flush_pg_stats: '[' 300 -eq 0 ']' 2019-12-21T16:07:45.358 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 0 2019-12-21T16:07:45.595 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 21474836494 -lt 21474836495 2019-12-21T16:07:45.595 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1991: flush_pg_stats: sleep 1 2019-12-21T16:07:46.596 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1992: flush_pg_stats: '[' 299 -eq 0 ']' 2019-12-21T16:07:46.597 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 0 2019-12-21T16:07:46.830 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 21474836496 -lt 21474836495 2019-12-21T16:07:46.831 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: flush_pg_stats: for s in $seqs 2019-12-21T16:07:46.831 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: echo 1-42949672974 2019-12-21T16:07:46.835 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: cut -d - -f 1 2019-12-21T16:07:46.837 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: osd=1 2019-12-21T16:07:46.837 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: echo 1-42949672974 2019-12-21T16:07:46.837 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: cut -d - -f 2 2019-12-21T16:07:46.837 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: seq=42949672974 2019-12-21T16:07:46.838 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1989: flush_pg_stats: echo 'waiting osd.1 seq 42949672974' 2019-12-21T16:07:46.838 INFO:tasks.workunit.client.0.smithi172.stderr:waiting osd.1 seq 42949672974 2019-12-21T16:07:46.838 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 1 2019-12-21T16:07:47.062 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 42949672974 -lt 42949672974 2019-12-21T16:07:47.062 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: flush_pg_stats: for s in $seqs 2019-12-21T16:07:47.062 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: echo 2-64424509453 2019-12-21T16:07:47.062 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: cut -d - -f 1 2019-12-21T16:07:47.063 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: osd=2 2019-12-21T16:07:47.064 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: echo 2-64424509453 2019-12-21T16:07:47.064 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: cut -d - -f 2 2019-12-21T16:07:47.065 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: seq=64424509453 2019-12-21T16:07:47.065 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1989: flush_pg_stats: echo 'waiting osd.2 seq 64424509453' 2019-12-21T16:07:47.065 INFO:tasks.workunit.client.0.smithi172.stderr:waiting osd.2 seq 64424509453 2019-12-21T16:07:47.065 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 2 2019-12-21T16:07:47.294 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 64424509453 -lt 64424509453 2019-12-21T16:07:47.294 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: flush_pg_stats: for s in $seqs 2019-12-21T16:07:47.295 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: echo 3-197568495618 2019-12-21T16:07:47.295 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: cut -d - -f 1 2019-12-21T16:07:47.296 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: osd=3 2019-12-21T16:07:47.296 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: echo 3-197568495618 2019-12-21T16:07:47.296 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: cut -d - -f 2 2019-12-21T16:07:47.297 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: seq=197568495618 2019-12-21T16:07:47.298 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1989: flush_pg_stats: echo 'waiting osd.3 seq 197568495618' 2019-12-21T16:07:47.298 INFO:tasks.workunit.client.0.smithi172.stderr:waiting osd.3 seq 197568495618 2019-12-21T16:07:47.298 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 3 2019-12-21T16:07:47.524 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 197568495618 -lt 197568495618 2019-12-21T16:07:47.525 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1555: wait_for_clean: get_num_pgs 2019-12-21T16:07:47.525 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: ceph --format json status 2019-12-21T16:07:47.525 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: jq .pgmap.num_pgs 2019-12-21T16:07:47.795 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1555: wait_for_clean: test 5 == 0 2019-12-21T16:07:47.796 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1559: wait_for_clean: true 2019-12-21T16:07:47.796 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1563: wait_for_clean: get_num_active_clean 2019-12-21T16:07:47.796 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: get_num_active_clean: local expression 2019-12-21T16:07:47.796 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2019-12-21T16:07:47.796 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("stale") | not)' 2019-12-21T16:07:47.796 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: ceph --format json pg dump pgs 2019-12-21T16:07:47.796 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2019-12-21T16:07:48.070 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1563: wait_for_clean: cur_active_clean=5 2019-12-21T16:07:48.071 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1564: wait_for_clean: get_num_pgs 2019-12-21T16:07:48.071 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: ceph --format json status 2019-12-21T16:07:48.071 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: jq .pgmap.num_pgs 2019-12-21T16:07:48.341 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1564: wait_for_clean: test 5 = 5 2019-12-21T16:07:48.342 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1564: wait_for_clean: break 2019-12-21T16:07:48.342 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: return 0 2019-12-21T16:07:48.342 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:249: corrupt_and_repair_one: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2019-12-21T16:07:56.771 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:250: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2019-12-21T16:07:56.773 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:267: corrupt_and_repair_erasure_coded: corrupt_and_repair_one td/osd-scrub-repair ecpool 1 2019-12-21T16:07:56.773 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:232: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2019-12-21T16:07:56.774 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:233: corrupt_and_repair_one: local poolname=ecpool 2019-12-21T16:07:56.774 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:234: corrupt_and_repair_one: local osd=1 2019-12-21T16:07:56.774 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:239: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2019-12-21T16:07:56.774 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: objectstore_tool: local dir=td/osd-scrub-repair 2019-12-21T16:07:56.774 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: objectstore_tool: shift 2019-12-21T16:07:56.775 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1235: objectstore_tool: local id=1 2019-12-21T16:07:56.775 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1236: objectstore_tool: shift 2019-12-21T16:07:56.775 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1238: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING remove 2019-12-21T16:07:56.775 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1205: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2019-12-21T16:07:56.775 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nowait: shift 2019-12-21T16:07:56.776 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1207: _objectstore_tool_nowait: local id=1 2019-12-21T16:07:56.776 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1208: _objectstore_tool_nowait: shift 2019-12-21T16:07:56.776 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1210: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2019-12-21T16:07:56.776 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2019-12-21T16:07:56.776 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2019-12-21T16:07:56.777 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2019-12-21T16:07:56.777 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2019-12-21T16:07:56.777 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2019-12-21T16:07:56.879 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2019-12-21T16:07:56.879 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1212: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING remove 2019-12-21T16:07:56.879 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1193: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2019-12-21T16:07:56.879 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1194: _objectstore_tool_nodown: shift 2019-12-21T16:07:56.880 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1195: _objectstore_tool_nodown: local id=1 2019-12-21T16:07:56.880 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: _objectstore_tool_nodown: shift 2019-12-21T16:07:56.880 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2019-12-21T16:07:56.880 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING remove 2019-12-21T16:07:57.588 INFO:tasks.workunit.client.0.smithi172.stdout:remove 1#2:eb822e21:::SOMETHING:head# 2019-12-21T16:07:58.107 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1213: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2019-12-21T16:07:58.107 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:842: activate_osd: local dir=td/osd-scrub-repair 2019-12-21T16:07:58.107 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: shift 2019-12-21T16:07:58.108 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: local id=1 2019-12-21T16:07:58.108 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: shift 2019-12-21T16:07:58.108 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local osd_data=td/osd-scrub-repair/1 2019-12-21T16:07:58.108 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local 'ceph_args=--fsid=973e3fc1-e63e-4bde-9db6-84c68fd5e93e --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2019-12-21T16:07:58.108 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2019-12-21T16:07:58.108 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-journal-size=100' 2019-12-21T16:07:58.108 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2019-12-21T16:07:58.109 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2019-12-21T16:07:58.109 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2019-12-21T16:07:58.109 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --chdir=' 2019-12-21T16:07:58.109 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+= 2019-12-21T16:07:58.109 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2019-12-21T16:07:58.109 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: get_asok_path 2019-12-21T16:07:58.109 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2019-12-21T16:07:58.110 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2019-12-21T16:07:58.110 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2019-12-21T16:07:58.110 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T16:07:58.110 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.119575 2019-12-21T16:07:58.110 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.119575/$cluster-$name.asok' 2019-12-21T16:07:58.110 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.119575/$cluster-$name.asok' 2019-12-21T16:07:58.111 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --debug-osd=20' 2019-12-21T16:07:58.111 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2019-12-21T16:07:58.111 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2019-12-21T16:07:58.111 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2019-12-21T16:07:58.111 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2019-12-21T16:07:58.111 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2019-12-21T16:07:58.111 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' ' 2019-12-21T16:07:58.112 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+= 2019-12-21T16:07:58.112 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: mkdir -p td/osd-scrub-repair/1 2019-12-21T16:07:58.112 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: echo start osd.1 2019-12-21T16:07:58.112 INFO:tasks.workunit.client.0.smithi172.stderr:start osd.1 2019-12-21T16:07:58.113 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph-osd -i 1 --fsid=973e3fc1-e63e-4bde-9db6-84c68fd5e93e --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.119575/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2019-12-21T16:07:58.113 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: cat td/osd-scrub-repair/1/whoami 2019-12-21T16:07:58.113 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: '[' 1 = 1 ']' 2019-12-21T16:07:58.113 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph osd dump --format=json 2019-12-21T16:07:58.114 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: jq '.flags_set[]' 2019-12-21T16:07:58.114 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: grep -q '"noup"' 2019-12-21T16:07:58.130 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:07:58.127+0000 7f0447916ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T16:07:58.132 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:07:58.129+0000 7f0447916ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T16:07:58.134 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:07:58.131+0000 7f0447916ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T16:07:58.349 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: wait_for_osd up 1 2019-12-21T16:07:58.349 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:914: wait_for_osd: local state=up 2019-12-21T16:07:58.349 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local id=1 2019-12-21T16:07:58.350 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: status=1 2019-12-21T16:07:58.350 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i=0 )) 2019-12-21T16:07:58.350 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i < 300 )) 2019-12-21T16:07:58.350 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: echo 0 2019-12-21T16:07:58.350 INFO:tasks.workunit.client.0.smithi172.stderr:0 2019-12-21T16:07:58.350 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: ceph osd dump 2019-12-21T16:07:58.350 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: grep 'osd.1 up' 2019-12-21T16:07:58.576 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: sleep 1 2019-12-21T16:07:58.649 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:07:58.646+0000 7f0447916ec0 -1 Falling back to public interface 2019-12-21T16:07:59.466 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:07:59.464+0000 7f0447916ec0 -1 osd.1 47 log_to_monitors {default=true} 2019-12-21T16:07:59.577 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i++ )) 2019-12-21T16:07:59.577 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i < 300 )) 2019-12-21T16:07:59.578 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: echo 1 2019-12-21T16:07:59.578 INFO:tasks.workunit.client.0.smithi172.stderr:1 2019-12-21T16:07:59.578 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: ceph osd dump 2019-12-21T16:07:59.578 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: grep 'osd.1 up' 2019-12-21T16:07:59.826 INFO:tasks.workunit.client.0.smithi172.stderr:osd.1 up in weight 1 up_from 50 up_thru 44 down_at 48 last_clean_interval [10,47) [v2:127.0.0.1:6810/170125,v1:127.0.0.1:6811/170125] [v2:127.0.0.1:6812/170125,v1:127.0.0.1:6813/170125] exists,up 356c69ab-d874-4bb4-813b-1608e502f8c0 2019-12-21T16:07:59.826 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=0 2019-12-21T16:07:59.827 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: break 2019-12-21T16:07:59.827 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: return 0 2019-12-21T16:07:59.827 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1239: objectstore_tool: wait_for_clean 2019-12-21T16:07:59.827 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1548: wait_for_clean: local cmd= 2019-12-21T16:07:59.827 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1549: wait_for_clean: local num_active_clean=-1 2019-12-21T16:07:59.827 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1550: wait_for_clean: local cur_active_clean 2019-12-21T16:07:59.828 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1551: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2019-12-21T16:07:59.828 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1551: wait_for_clean: get_timeout_delays 90 .1 2019-12-21T16:07:59.828 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1494: get_timeout_delays: shopt -q -o xtrace 2019-12-21T16:07:59.828 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1494: get_timeout_delays: echo true 2019-12-21T16:07:59.828 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1494: get_timeout_delays: local trace=true 2019-12-21T16:07:59.828 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1495: get_timeout_delays: true 2019-12-21T16:07:59.829 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1495: get_timeout_delays: shopt -u -o xtrace 2019-12-21T16:07:59.919 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1551: wait_for_clean: local -a delays 2019-12-21T16:07:59.919 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1552: wait_for_clean: local -i loop=0 2019-12-21T16:07:59.919 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1554: wait_for_clean: flush_pg_stats 2019-12-21T16:07:59.920 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1977: flush_pg_stats: local timeout=300 2019-12-21T16:07:59.920 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1979: flush_pg_stats: ceph osd ls 2019-12-21T16:08:00.210 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1979: flush_pg_stats: ids='0 2019-12-21T16:08:00.211 INFO:tasks.workunit.client.0.smithi172.stderr:1 2019-12-21T16:08:00.211 INFO:tasks.workunit.client.0.smithi172.stderr:2 2019-12-21T16:08:00.211 INFO:tasks.workunit.client.0.smithi172.stderr:3' 2019-12-21T16:08:00.211 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1980: flush_pg_stats: seqs= 2019-12-21T16:08:00.211 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: flush_pg_stats: for osd in $ids 2019-12-21T16:08:00.211 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2019-12-21T16:08:00.322 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: seq=21474836499 2019-12-21T16:08:00.322 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: flush_pg_stats: seqs=' 0-21474836499' 2019-12-21T16:08:00.322 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: flush_pg_stats: for osd in $ids 2019-12-21T16:08:00.322 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2019-12-21T16:08:00.447 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: seq=214748364802 2019-12-21T16:08:00.447 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: flush_pg_stats: seqs=' 0-21474836499 1-214748364802' 2019-12-21T16:08:00.448 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: flush_pg_stats: for osd in $ids 2019-12-21T16:08:00.448 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2019-12-21T16:08:00.560 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: seq=64424509457 2019-12-21T16:08:00.560 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: flush_pg_stats: seqs=' 0-21474836499 1-214748364802 2-64424509457' 2019-12-21T16:08:00.560 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: flush_pg_stats: for osd in $ids 2019-12-21T16:08:00.561 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2019-12-21T16:08:00.670 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: seq=197568495622 2019-12-21T16:08:00.670 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: flush_pg_stats: seqs=' 0-21474836499 1-214748364802 2-64424509457 3-197568495622' 2019-12-21T16:08:00.671 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: flush_pg_stats: for s in $seqs 2019-12-21T16:08:00.671 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: echo 0-21474836499 2019-12-21T16:08:00.671 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: cut -d - -f 1 2019-12-21T16:08:00.672 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: osd=0 2019-12-21T16:08:00.672 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: echo 0-21474836499 2019-12-21T16:08:00.672 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: cut -d - -f 2 2019-12-21T16:08:00.673 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: seq=21474836499 2019-12-21T16:08:00.673 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1989: flush_pg_stats: echo 'waiting osd.0 seq 21474836499' 2019-12-21T16:08:00.673 INFO:tasks.workunit.client.0.smithi172.stderr:waiting osd.0 seq 21474836499 2019-12-21T16:08:00.674 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 0 2019-12-21T16:08:00.901 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 21474836498 -lt 21474836499 2019-12-21T16:08:00.901 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1991: flush_pg_stats: sleep 1 2019-12-21T16:08:01.902 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1992: flush_pg_stats: '[' 300 -eq 0 ']' 2019-12-21T16:08:01.902 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 0 2019-12-21T16:08:02.139 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 21474836500 -lt 21474836499 2019-12-21T16:08:02.139 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: flush_pg_stats: for s in $seqs 2019-12-21T16:08:02.140 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: echo 1-214748364802 2019-12-21T16:08:02.140 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: cut -d - -f 1 2019-12-21T16:08:02.140 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: osd=1 2019-12-21T16:08:02.141 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: echo 1-214748364802 2019-12-21T16:08:02.141 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: cut -d - -f 2 2019-12-21T16:08:02.142 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: seq=214748364802 2019-12-21T16:08:02.142 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1989: flush_pg_stats: echo 'waiting osd.1 seq 214748364802' 2019-12-21T16:08:02.143 INFO:tasks.workunit.client.0.smithi172.stderr:waiting osd.1 seq 214748364802 2019-12-21T16:08:02.143 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 1 2019-12-21T16:08:02.370 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 214748364802 -lt 214748364802 2019-12-21T16:08:02.370 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: flush_pg_stats: for s in $seqs 2019-12-21T16:08:02.371 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: echo 2-64424509457 2019-12-21T16:08:02.371 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: cut -d - -f 1 2019-12-21T16:08:02.371 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: osd=2 2019-12-21T16:08:02.372 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: echo 2-64424509457 2019-12-21T16:08:02.372 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: cut -d - -f 2 2019-12-21T16:08:02.373 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: seq=64424509457 2019-12-21T16:08:02.373 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1989: flush_pg_stats: echo 'waiting osd.2 seq 64424509457' 2019-12-21T16:08:02.373 INFO:tasks.workunit.client.0.smithi172.stderr:waiting osd.2 seq 64424509457 2019-12-21T16:08:02.374 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 2 2019-12-21T16:08:02.600 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 64424509457 -lt 64424509457 2019-12-21T16:08:02.601 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: flush_pg_stats: for s in $seqs 2019-12-21T16:08:02.601 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: echo 3-197568495622 2019-12-21T16:08:02.601 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: cut -d - -f 1 2019-12-21T16:08:02.602 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: osd=3 2019-12-21T16:08:02.602 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: echo 3-197568495622 2019-12-21T16:08:02.603 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: cut -d - -f 2 2019-12-21T16:08:02.603 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: seq=197568495622 2019-12-21T16:08:02.604 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1989: flush_pg_stats: echo 'waiting osd.3 seq 197568495622' 2019-12-21T16:08:02.604 INFO:tasks.workunit.client.0.smithi172.stderr:waiting osd.3 seq 197568495622 2019-12-21T16:08:02.604 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 3 2019-12-21T16:08:02.830 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 197568495622 -lt 197568495622 2019-12-21T16:08:02.831 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1555: wait_for_clean: get_num_pgs 2019-12-21T16:08:02.831 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: ceph --format json status 2019-12-21T16:08:02.831 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: jq .pgmap.num_pgs 2019-12-21T16:08:03.100 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1555: wait_for_clean: test 5 == 0 2019-12-21T16:08:03.100 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1559: wait_for_clean: true 2019-12-21T16:08:03.100 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1563: wait_for_clean: get_num_active_clean 2019-12-21T16:08:03.101 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: get_num_active_clean: local expression 2019-12-21T16:08:03.101 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2019-12-21T16:08:03.101 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("stale") | not)' 2019-12-21T16:08:03.101 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: ceph --format json pg dump pgs 2019-12-21T16:08:03.101 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2019-12-21T16:08:03.324 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1563: wait_for_clean: cur_active_clean=5 2019-12-21T16:08:03.324 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1564: wait_for_clean: get_num_pgs 2019-12-21T16:08:03.324 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: ceph --format json status 2019-12-21T16:08:03.325 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: jq .pgmap.num_pgs 2019-12-21T16:08:03.596 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1564: wait_for_clean: test 5 = 5 2019-12-21T16:08:03.597 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1564: wait_for_clean: break 2019-12-21T16:08:03.597 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: return 0 2019-12-21T16:08:03.597 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:243: corrupt_and_repair_one: get_pg ecpool SOMETHING 2019-12-21T16:08:03.597 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1029: get_pg: local poolname=ecpool 2019-12-21T16:08:03.597 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1030: get_pg: local objectname=SOMETHING 2019-12-21T16:08:03.597 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1032: get_pg: ceph --format json osd map ecpool SOMETHING 2019-12-21T16:08:03.598 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1032: get_pg: jq -r .pgid 2019-12-21T16:08:03.830 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:243: corrupt_and_repair_one: local pg=2.0 2019-12-21T16:08:03.830 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:244: corrupt_and_repair_one: repair 2.0 2019-12-21T16:08:03.830 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: repair: local pgid=2.0 2019-12-21T16:08:03.830 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1665: repair: get_last_scrub_stamp 2.0 2019-12-21T16:08:03.830 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=2.0 2019-12-21T16:08:03.831 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T16:08:03.831 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T16:08:03.831 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2019-12-21T16:08:04.052 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1665: repair: local last_scrub=2019-12-21T16:07:34.952470+0000 2019-12-21T16:08:04.052 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1666: repair: ceph pg repair 2.0 2019-12-21T16:08:04.267 INFO:tasks.workunit.client.0.smithi172.stderr:instructing pg 2.0s0 on osd.3 to repair 2019-12-21T16:08:04.276 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: repair: wait_for_scrub 2.0 2019-12-21T16:07:34.952470+0000 2019-12-21T16:08:04.276 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1789: wait_for_scrub: local pgid=2.0 2019-12-21T16:08:04.276 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1790: wait_for_scrub: local last_scrub=2019-12-21T16:07:34.952470+0000 2019-12-21T16:08:04.277 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1791: wait_for_scrub: local sname=last_scrub_stamp 2019-12-21T16:08:04.277 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i=0 )) 2019-12-21T16:08:04.277 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T16:08:04.277 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2019-12-21T16:08:04.277 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=2.0 2019-12-21T16:08:04.277 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T16:08:04.277 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T16:08:04.278 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2019-12-21T16:08:04.498 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T16:07:34.952470+0000 '>' 2019-12-21T16:07:34.952470+0000 2019-12-21T16:08:04.498 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T16:08:05.499 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T16:08:05.499 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T16:08:05.500 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2019-12-21T16:08:05.500 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=2.0 2019-12-21T16:08:05.501 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T16:08:05.501 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T16:08:05.501 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2019-12-21T16:08:05.734 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T16:07:34.952470+0000 '>' 2019-12-21T16:07:34.952470+0000 2019-12-21T16:08:05.734 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T16:08:06.735 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T16:08:06.735 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T16:08:06.736 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2019-12-21T16:08:06.736 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=2.0 2019-12-21T16:08:06.737 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T16:08:06.737 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T16:08:06.737 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2019-12-21T16:08:06.968 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T16:07:34.952470+0000 '>' 2019-12-21T16:07:34.952470+0000 2019-12-21T16:08:06.969 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T16:08:07.970 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T16:08:07.970 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T16:08:07.971 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2019-12-21T16:08:07.971 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=2.0 2019-12-21T16:08:07.971 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T16:08:07.971 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T16:08:07.971 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2019-12-21T16:08:08.203 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T16:07:34.952470+0000 '>' 2019-12-21T16:07:34.952470+0000 2019-12-21T16:08:08.203 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T16:08:09.204 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T16:08:09.204 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T16:08:09.205 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2019-12-21T16:08:09.205 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=2.0 2019-12-21T16:08:09.206 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T16:08:09.206 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T16:08:09.206 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2019-12-21T16:08:09.437 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T16:07:34.952470+0000 '>' 2019-12-21T16:07:34.952470+0000 2019-12-21T16:08:09.437 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_scrub: sleep 1 2019-12-21T16:08:10.438 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i++ )) 2019-12-21T16:08:10.438 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: wait_for_scrub: (( i < 300 )) 2019-12-21T16:08:10.439 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2019-12-21T16:08:10.439 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: get_last_scrub_stamp: local pgid=2.0 2019-12-21T16:08:10.439 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: get_last_scrub_stamp: local sname=last_scrub_stamp 2019-12-21T16:08:10.439 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: get_last_scrub_stamp: ceph --format json pg dump pgs 2019-12-21T16:08:10.439 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2019-12-21T16:08:10.674 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1794: wait_for_scrub: test 2019-12-21T16:08:05.086197+0000 '>' 2019-12-21T16:07:34.952470+0000 2019-12-21T16:08:10.674 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1795: wait_for_scrub: return 0 2019-12-21T16:08:10.675 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:248: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2019-12-21T16:08:10.675 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: objectstore_tool: local dir=td/osd-scrub-repair 2019-12-21T16:08:10.675 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: objectstore_tool: shift 2019-12-21T16:08:10.675 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1235: objectstore_tool: local id=3 2019-12-21T16:08:10.675 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1236: objectstore_tool: shift 2019-12-21T16:08:10.675 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1238: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING list-attrs 2019-12-21T16:08:10.676 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1205: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2019-12-21T16:08:10.676 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nowait: shift 2019-12-21T16:08:10.676 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1207: _objectstore_tool_nowait: local id=3 2019-12-21T16:08:10.676 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1208: _objectstore_tool_nowait: shift 2019-12-21T16:08:10.676 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1210: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2019-12-21T16:08:10.676 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2019-12-21T16:08:10.676 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2019-12-21T16:08:10.677 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2019-12-21T16:08:10.677 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2019-12-21T16:08:10.677 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2019-12-21T16:08:10.779 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2019-12-21T16:08:10.779 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1212: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING list-attrs 2019-12-21T16:08:10.779 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1193: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2019-12-21T16:08:10.779 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1194: _objectstore_tool_nodown: shift 2019-12-21T16:08:10.779 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1195: _objectstore_tool_nodown: local id=3 2019-12-21T16:08:10.779 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: _objectstore_tool_nodown: shift 2019-12-21T16:08:10.780 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2019-12-21T16:08:10.780 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING list-attrs 2019-12-21T16:08:11.373 INFO:tasks.workunit.client.0.smithi172.stderr:Error getting attr on : 2.0s0_head,0#-4:00000000:::scrub_2.0s0:head#, (61) No data available 2019-12-21T16:08:11.373 INFO:tasks.workunit.client.0.smithi172.stdout:_ 2019-12-21T16:08:11.373 INFO:tasks.workunit.client.0.smithi172.stdout:hinfo_key 2019-12-21T16:08:11.373 INFO:tasks.workunit.client.0.smithi172.stdout:snapset 2019-12-21T16:08:11.890 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1213: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2019-12-21T16:08:11.890 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:842: activate_osd: local dir=td/osd-scrub-repair 2019-12-21T16:08:11.891 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: shift 2019-12-21T16:08:11.891 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: local id=3 2019-12-21T16:08:11.891 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: shift 2019-12-21T16:08:11.891 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local osd_data=td/osd-scrub-repair/3 2019-12-21T16:08:11.891 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local 'ceph_args=--fsid=973e3fc1-e63e-4bde-9db6-84c68fd5e93e --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2019-12-21T16:08:11.891 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2019-12-21T16:08:11.892 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-journal-size=100' 2019-12-21T16:08:11.892 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2019-12-21T16:08:11.892 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2019-12-21T16:08:11.892 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2019-12-21T16:08:11.892 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --chdir=' 2019-12-21T16:08:11.892 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+= 2019-12-21T16:08:11.892 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2019-12-21T16:08:11.893 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: get_asok_path 2019-12-21T16:08:11.893 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2019-12-21T16:08:11.893 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2019-12-21T16:08:11.893 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2019-12-21T16:08:11.893 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T16:08:11.893 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.119575 2019-12-21T16:08:11.893 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.119575/$cluster-$name.asok' 2019-12-21T16:08:11.895 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.119575/$cluster-$name.asok' 2019-12-21T16:08:11.895 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --debug-osd=20' 2019-12-21T16:08:11.895 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2019-12-21T16:08:11.896 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2019-12-21T16:08:11.896 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2019-12-21T16:08:11.896 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2019-12-21T16:08:11.896 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2019-12-21T16:08:11.896 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' ' 2019-12-21T16:08:11.896 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+= 2019-12-21T16:08:11.897 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: mkdir -p td/osd-scrub-repair/3 2019-12-21T16:08:11.897 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: echo start osd.3 2019-12-21T16:08:11.897 INFO:tasks.workunit.client.0.smithi172.stderr:start osd.3 2019-12-21T16:08:11.897 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph-osd -i 3 --fsid=973e3fc1-e63e-4bde-9db6-84c68fd5e93e --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.119575/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2019-12-21T16:08:11.897 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: cat td/osd-scrub-repair/3/whoami 2019-12-21T16:08:11.897 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: '[' 3 = 3 ']' 2019-12-21T16:08:11.897 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph osd dump --format=json 2019-12-21T16:08:11.898 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: jq '.flags_set[]' 2019-12-21T16:08:11.898 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: grep -q '"noup"' 2019-12-21T16:08:11.914 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:08:11.912+0000 7f0f16016ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T16:08:11.916 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:08:11.914+0000 7f0f16016ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T16:08:11.919 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:08:11.917+0000 7f0f16016ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T16:08:12.134 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: wait_for_osd up 3 2019-12-21T16:08:12.134 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:914: wait_for_osd: local state=up 2019-12-21T16:08:12.135 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local id=3 2019-12-21T16:08:12.135 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: status=1 2019-12-21T16:08:12.135 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i=0 )) 2019-12-21T16:08:12.135 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i < 300 )) 2019-12-21T16:08:12.135 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: echo 0 2019-12-21T16:08:12.135 INFO:tasks.workunit.client.0.smithi172.stderr:0 2019-12-21T16:08:12.136 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: ceph osd dump 2019-12-21T16:08:12.136 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: grep 'osd.3 up' 2019-12-21T16:08:12.362 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: sleep 1 2019-12-21T16:08:12.434 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:08:12.433+0000 7f0f16016ec0 -1 Falling back to public interface 2019-12-21T16:08:13.117 INFO:tasks.workunit.client.0.smithi172.stderr:2019-12-21T16:08:13.115+0000 7f0f16016ec0 -1 osd.3 52 log_to_monitors {default=true} 2019-12-21T16:08:13.363 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i++ )) 2019-12-21T16:08:13.363 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i < 300 )) 2019-12-21T16:08:13.363 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: echo 1 2019-12-21T16:08:13.364 INFO:tasks.workunit.client.0.smithi172.stderr:1 2019-12-21T16:08:13.364 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: grep 'osd.3 up' 2019-12-21T16:08:13.364 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: ceph osd dump 2019-12-21T16:08:13.661 INFO:tasks.workunit.client.0.smithi172.stderr:osd.3 up in weight 1 up_from 55 up_thru 55 down_at 53 last_clean_interval [46,52) [v2:127.0.0.1:6826/171088,v1:127.0.0.1:6827/171088] [v2:127.0.0.1:6828/171088,v1:127.0.0.1:6829/171088] exists,up 245349d5-5742-4af0-b9e3-1b7e828df542 2019-12-21T16:08:13.662 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=0 2019-12-21T16:08:13.662 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: break 2019-12-21T16:08:13.662 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: return 0 2019-12-21T16:08:13.662 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1239: objectstore_tool: wait_for_clean 2019-12-21T16:08:13.662 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1548: wait_for_clean: local cmd= 2019-12-21T16:08:13.662 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1549: wait_for_clean: local num_active_clean=-1 2019-12-21T16:08:13.663 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1550: wait_for_clean: local cur_active_clean 2019-12-21T16:08:13.663 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1551: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2019-12-21T16:08:13.663 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1551: wait_for_clean: get_timeout_delays 90 .1 2019-12-21T16:08:13.663 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1494: get_timeout_delays: shopt -q -o xtrace 2019-12-21T16:08:13.663 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1494: get_timeout_delays: echo true 2019-12-21T16:08:13.663 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1494: get_timeout_delays: local trace=true 2019-12-21T16:08:13.663 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1495: get_timeout_delays: true 2019-12-21T16:08:13.664 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1495: get_timeout_delays: shopt -u -o xtrace 2019-12-21T16:08:13.753 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1551: wait_for_clean: local -a delays 2019-12-21T16:08:13.753 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1552: wait_for_clean: local -i loop=0 2019-12-21T16:08:13.753 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1554: wait_for_clean: flush_pg_stats 2019-12-21T16:08:13.753 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1977: flush_pg_stats: local timeout=300 2019-12-21T16:08:13.754 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1979: flush_pg_stats: ceph osd ls 2019-12-21T16:08:14.053 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1979: flush_pg_stats: ids='0 2019-12-21T16:08:14.053 INFO:tasks.workunit.client.0.smithi172.stderr:1 2019-12-21T16:08:14.053 INFO:tasks.workunit.client.0.smithi172.stderr:2 2019-12-21T16:08:14.053 INFO:tasks.workunit.client.0.smithi172.stderr:3' 2019-12-21T16:08:14.054 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1980: flush_pg_stats: seqs= 2019-12-21T16:08:14.054 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: flush_pg_stats: for osd in $ids 2019-12-21T16:08:14.054 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2019-12-21T16:08:14.164 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: seq=21474836503 2019-12-21T16:08:14.164 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: flush_pg_stats: seqs=' 0-21474836503' 2019-12-21T16:08:14.164 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: flush_pg_stats: for osd in $ids 2019-12-21T16:08:14.165 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2019-12-21T16:08:14.276 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: seq=214748364805 2019-12-21T16:08:14.276 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: flush_pg_stats: seqs=' 0-21474836503 1-214748364805' 2019-12-21T16:08:14.276 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: flush_pg_stats: for osd in $ids 2019-12-21T16:08:14.276 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2019-12-21T16:08:14.408 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: seq=64424509461 2019-12-21T16:08:14.408 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: flush_pg_stats: seqs=' 0-21474836503 1-214748364805 2-64424509461' 2019-12-21T16:08:14.408 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: flush_pg_stats: for osd in $ids 2019-12-21T16:08:14.408 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2019-12-21T16:08:14.518 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: flush_pg_stats: seq=236223201282 2019-12-21T16:08:14.519 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: flush_pg_stats: seqs=' 0-21474836503 1-214748364805 2-64424509461 3-236223201282' 2019-12-21T16:08:14.519 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: flush_pg_stats: for s in $seqs 2019-12-21T16:08:14.519 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: echo 0-21474836503 2019-12-21T16:08:14.519 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: cut -d - -f 1 2019-12-21T16:08:14.520 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: osd=0 2019-12-21T16:08:14.520 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: echo 0-21474836503 2019-12-21T16:08:14.521 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: cut -d - -f 2 2019-12-21T16:08:14.521 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: seq=21474836503 2019-12-21T16:08:14.521 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1989: flush_pg_stats: echo 'waiting osd.0 seq 21474836503' 2019-12-21T16:08:14.522 INFO:tasks.workunit.client.0.smithi172.stderr:waiting osd.0 seq 21474836503 2019-12-21T16:08:14.522 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 0 2019-12-21T16:08:14.748 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 21474836502 -lt 21474836503 2019-12-21T16:08:14.748 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1991: flush_pg_stats: sleep 1 2019-12-21T16:08:15.749 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1992: flush_pg_stats: '[' 300 -eq 0 ']' 2019-12-21T16:08:15.750 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 0 2019-12-21T16:08:15.986 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 21474836504 -lt 21474836503 2019-12-21T16:08:15.986 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: flush_pg_stats: for s in $seqs 2019-12-21T16:08:15.987 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: echo 1-214748364805 2019-12-21T16:08:15.987 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: cut -d - -f 1 2019-12-21T16:08:15.987 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: osd=1 2019-12-21T16:08:15.988 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: echo 1-214748364805 2019-12-21T16:08:15.988 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: cut -d - -f 2 2019-12-21T16:08:15.989 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: seq=214748364805 2019-12-21T16:08:15.989 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1989: flush_pg_stats: echo 'waiting osd.1 seq 214748364805' 2019-12-21T16:08:15.989 INFO:tasks.workunit.client.0.smithi172.stderr:waiting osd.1 seq 214748364805 2019-12-21T16:08:15.990 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 1 2019-12-21T16:08:16.229 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 214748364806 -lt 214748364805 2019-12-21T16:08:16.229 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: flush_pg_stats: for s in $seqs 2019-12-21T16:08:16.230 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: echo 2-64424509461 2019-12-21T16:08:16.230 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: cut -d - -f 1 2019-12-21T16:08:16.230 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: osd=2 2019-12-21T16:08:16.231 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: echo 2-64424509461 2019-12-21T16:08:16.231 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: cut -d - -f 2 2019-12-21T16:08:16.232 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: seq=64424509461 2019-12-21T16:08:16.232 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1989: flush_pg_stats: echo 'waiting osd.2 seq 64424509461' 2019-12-21T16:08:16.232 INFO:tasks.workunit.client.0.smithi172.stderr:waiting osd.2 seq 64424509461 2019-12-21T16:08:16.233 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 2 2019-12-21T16:08:16.459 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 64424509461 -lt 64424509461 2019-12-21T16:08:16.459 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: flush_pg_stats: for s in $seqs 2019-12-21T16:08:16.460 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: echo 3-236223201282 2019-12-21T16:08:16.460 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: cut -d - -f 1 2019-12-21T16:08:16.460 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: flush_pg_stats: osd=3 2019-12-21T16:08:16.463 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: echo 3-236223201282 2019-12-21T16:08:16.463 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: cut -d - -f 2 2019-12-21T16:08:16.463 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: flush_pg_stats: seq=236223201282 2019-12-21T16:08:16.463 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1989: flush_pg_stats: echo 'waiting osd.3 seq 236223201282' 2019-12-21T16:08:16.463 INFO:tasks.workunit.client.0.smithi172.stderr:waiting osd.3 seq 236223201282 2019-12-21T16:08:16.463 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: ceph osd last-stat-seq 3 2019-12-21T16:08:16.690 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: flush_pg_stats: test 236223201282 -lt 236223201282 2019-12-21T16:08:16.690 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1555: wait_for_clean: get_num_pgs 2019-12-21T16:08:16.691 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: ceph --format json status 2019-12-21T16:08:16.691 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: jq .pgmap.num_pgs 2019-12-21T16:08:16.981 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1555: wait_for_clean: test 5 == 0 2019-12-21T16:08:16.981 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1559: wait_for_clean: true 2019-12-21T16:08:16.983 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1563: wait_for_clean: get_num_active_clean 2019-12-21T16:08:16.983 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: get_num_active_clean: local expression 2019-12-21T16:08:16.983 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2019-12-21T16:08:16.983 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("stale") | not)' 2019-12-21T16:08:16.983 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: ceph --format json pg dump pgs 2019-12-21T16:08:16.983 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2019-12-21T16:08:17.204 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1563: wait_for_clean: cur_active_clean=5 2019-12-21T16:08:17.204 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1564: wait_for_clean: get_num_pgs 2019-12-21T16:08:17.204 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: ceph --format json status 2019-12-21T16:08:17.205 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_num_pgs: jq .pgmap.num_pgs 2019-12-21T16:08:17.473 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1564: wait_for_clean: test 5 = 5 2019-12-21T16:08:17.473 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1564: wait_for_clean: break 2019-12-21T16:08:17.473 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: return 0 2019-12-21T16:08:17.473 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:249: corrupt_and_repair_one: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2019-12-21T16:08:27.129 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:250: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2019-12-21T16:08:27.132 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:268: corrupt_and_repair_erasure_coded: corrupt_and_repair_two td/osd-scrub-repair ecpool 1 2 2019-12-21T16:08:27.133 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:191: corrupt_and_repair_two: local dir=td/osd-scrub-repair 2019-12-21T16:08:27.133 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:192: corrupt_and_repair_two: local poolname=ecpool 2019-12-21T16:08:27.133 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:193: corrupt_and_repair_two: local first=1 2019-12-21T16:08:27.133 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:194: corrupt_and_repair_two: local second=2 2019-12-21T16:08:27.133 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:199: corrupt_and_repair_two: pids= 2019-12-21T16:08:27.133 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:200: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2019-12-21T16:08:27.134 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1907: run_in_background: local pid_variable=pids 2019-12-21T16:08:27.134 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1908: run_in_background: shift 2019-12-21T16:08:27.134 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1912: run_in_background: eval 'pids+=" 171746"' 2019-12-21T16:08:27.134 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1912: run_in_background: pids+=' 171746' 2019-12-21T16:08:27.134 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:201: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 2 SOMETHING remove 2019-12-21T16:08:27.134 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1907: run_in_background: local pid_variable=pids 2019-12-21T16:08:27.134 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1908: run_in_background: shift 2019-12-21T16:08:27.134 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1912: run_in_background: eval 'pids+=" 171748"' 2019-12-21T16:08:27.135 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1912: run_in_background: pids+=' 171748' 2019-12-21T16:08:27.135 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1911: run_in_background: objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2019-12-21T16:08:27.135 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:202: corrupt_and_repair_two: wait_background pids 2019-12-21T16:08:27.135 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1943: wait_background: pids=' 171746 171748' 2019-12-21T16:08:27.135 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1945: wait_background: return_code=0 2019-12-21T16:08:27.135 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1946: wait_background: for pid in $pids 2019-12-21T16:08:27.135 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1947: wait_background: wait 171746 2019-12-21T16:08:27.135 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1911: run_in_background: sed 's/^/119575: /' 2019-12-21T16:08:27.136 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1911: run_in_background: objectstore_tool td/osd-scrub-repair 2 SOMETHING remove 2019-12-21T16:08:27.136 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1911: run_in_background: sed 's/^/119575: /' 2019-12-21T16:08:28.649 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: objectstore_tool: local dir=td/osd-scrub-repair 2019-12-21T16:08:28.649 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: objectstore_tool: shift 2019-12-21T16:08:28.649 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1235: objectstore_tool: local id=1 2019-12-21T16:08:28.650 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1236: objectstore_tool: shift 2019-12-21T16:08:28.650 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1238: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING remove 2019-12-21T16:08:28.650 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1205: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2019-12-21T16:08:28.650 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nowait: shift 2019-12-21T16:08:28.650 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1207: _objectstore_tool_nowait: local id=1 2019-12-21T16:08:28.650 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1208: _objectstore_tool_nowait: shift 2019-12-21T16:08:28.651 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1210: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2019-12-21T16:08:28.651 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2019-12-21T16:08:28.651 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2019-12-21T16:08:28.651 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2019-12-21T16:08:28.651 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2019-12-21T16:08:28.651 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2019-12-21T16:08:28.652 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2019-12-21T16:08:28.652 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1212: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING remove 2019-12-21T16:08:28.652 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1193: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2019-12-21T16:08:28.652 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1194: _objectstore_tool_nodown: shift 2019-12-21T16:08:28.652 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1195: _objectstore_tool_nodown: local id=1 2019-12-21T16:08:28.652 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: _objectstore_tool_nodown: shift 2019-12-21T16:08:28.653 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2019-12-21T16:08:28.653 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING remove 2019-12-21T16:08:28.653 INFO:tasks.workunit.client.0.smithi172.stderr:119575: remove 1#2:eb822e21:::SOMETHING:head# 2019-12-21T16:08:28.653 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1213: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2019-12-21T16:08:28.653 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:842: activate_osd: local dir=td/osd-scrub-repair 2019-12-21T16:08:28.653 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: shift 2019-12-21T16:08:28.654 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: local id=1 2019-12-21T16:08:28.654 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: shift 2019-12-21T16:08:28.654 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local osd_data=td/osd-scrub-repair/1 2019-12-21T16:08:28.654 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local 'ceph_args=--fsid=973e3fc1-e63e-4bde-9db6-84c68fd5e93e --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2019-12-21T16:08:28.654 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2019-12-21T16:08:28.655 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-journal-size=100' 2019-12-21T16:08:28.655 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2019-12-21T16:08:28.655 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2019-12-21T16:08:28.655 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2019-12-21T16:08:28.655 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --chdir=' 2019-12-21T16:08:28.655 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+= 2019-12-21T16:08:28.656 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2019-12-21T16:08:28.656 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: get_asok_path 2019-12-21T16:08:28.656 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2019-12-21T16:08:28.656 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2019-12-21T16:08:28.656 INFO:tasks.workunit.client.0.smithi172.stderr:119575: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2019-12-21T16:08:28.656 INFO:tasks.workunit.client.0.smithi172.stderr:119575: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T16:08:28.657 INFO:tasks.workunit.client.0.smithi172.stderr:119575: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.119575 2019-12-21T16:08:28.657 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.119575/$cluster-$name.asok' 2019-12-21T16:08:28.657 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.119575/$cluster-$name.asok' 2019-12-21T16:08:28.657 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --debug-osd=20' 2019-12-21T16:08:28.657 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2019-12-21T16:08:28.657 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2019-12-21T16:08:28.658 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2019-12-21T16:08:28.658 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2019-12-21T16:08:28.658 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2019-12-21T16:08:28.658 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' ' 2019-12-21T16:08:28.658 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+= 2019-12-21T16:08:28.658 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: mkdir -p td/osd-scrub-repair/1 2019-12-21T16:08:28.659 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: echo start osd.1 2019-12-21T16:08:28.659 INFO:tasks.workunit.client.0.smithi172.stderr:119575: start osd.1 2019-12-21T16:08:28.659 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph-osd -i 1 --fsid=973e3fc1-e63e-4bde-9db6-84c68fd5e93e --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.119575/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2019-12-21T16:08:28.659 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: cat td/osd-scrub-repair/1/whoami 2019-12-21T16:08:28.688 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standa119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: objectstore_tool: local dir=td/osd-scrub-repair 2019-12-21T16:08:28.688 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: objectstore_tool: shift 2019-12-21T16:08:28.688 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1235: objectstore_tool: local id=2 2019-12-21T16:08:28.688 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1236: objectstore_tool: shift 2019-12-21T16:08:28.689 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1238: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 2 SOMETHING remove 2019-12-21T16:08:28.690 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1205: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2019-12-21T16:08:28.690 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nowait: shift 2019-12-21T16:08:28.690 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1207: _objectstore_tool_nowait: local id=2 2019-12-21T16:08:28.690 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1208: _objectstore_tool_nowait: shift 2019-12-21T16:08:28.690 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1210: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.2 2019-12-21T16:08:28.690 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2019-12-21T16:08:28.691 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2019-12-21T16:08:28.691 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2019-12-21T16:08:28.691 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2019-12-21T16:08:28.691 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2019-12-21T16:08:28.691 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2019-12-21T16:08:28.691 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1212: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 2 SOMETHING remove 2019-12-21T16:08:28.692 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1193: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2019-12-21T16:08:28.692 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1194: _objectstore_tool_nodown: shift 2019-12-21T16:08:28.692 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1195: _objectstore_tool_nodown: local id=2 2019-12-21T16:08:28.693 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: _objectstore_tool_nodown: shift 2019-12-21T16:08:28.693 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/2 2019-12-21T16:08:28.693 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 SOMETHING remove 2019-12-21T16:08:28.693 INFO:tasks.workunit.client.0.smithi172.stderr:119575: remove 2#2:eb822e21:::SOMETHING:head# 2019-12-21T16:08:28.693 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1213: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 2 2019-12-21T16:08:28.693 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:842: activate_osd: local dir=td/osd-scrub-repair 2019-12-21T16:08:28.694 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: shift 2019-12-21T16:08:28.694 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: local id=2 2019-12-21T16:08:28.694 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: shift 2019-12-21T16:08:28.694 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local osd_data=td/osd-scrub-repair/2 2019-12-21T16:08:28.694 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local 'ceph_args=--fsid=973e3fc1-e63e-4bde-9db6-84c68fd5e93e --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2019-12-21T16:08:28.694 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2019-12-21T16:08:28.695 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-journal-size=100' 2019-12-21T16:08:28.695 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2019-12-21T16:08:28.695 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2019-12-21T16:08:28.695 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2019-12-21T16:08:28.696 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --chdir=' 2019-12-21T16:08:28.696 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+= 2019-12-21T16:08:28.696 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2019-12-21T16:08:28.696 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: get_asok_path 2019-12-21T16:08:28.696 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2019-12-21T16:08:28.696 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2019-12-21T16:08:28.697 INFO:tasks.workunit.client.0.smithi172.stderr:119575: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2019-12-21T16:08:28.697 INFO:tasks.workunit.client.0.smithi172.stderr:119575: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2019-12-21T16:08:28.697 INFO:tasks.workunit.client.0.smithi172.stderr:119575: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.119575 2019-12-21T16:08:28.697 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.119575/$cluster-$name.asok' 2019-12-21T16:08:28.697 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.119575/$cluster-$name.asok' 2019-12-21T16:08:28.697 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --debug-osd=20' 2019-12-21T16:08:28.698 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2019-12-21T16:08:28.698 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2019-12-21T16:08:28.698 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2019-12-21T16:08:28.698 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2019-12-21T16:08:28.698 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2019-12-21T16:08:28.698 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' ' 2019-12-21T16:08:28.698 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+= 2019-12-21T16:08:28.699 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: mkdir -p td/osd-scrub-repair/2 2019-12-21T16:08:28.699 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: echo start osd.2 2019-12-21T16:08:28.699 INFO:tasks.workunit.client.0.smithi172.stderr:119575: start osd.2 2019-12-21T16:08:28.699 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph-osd -i 2 --fsid=973e3fc1-e63e-4bde-9db6-84c68fd5e93e --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.119575/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2019-12-21T16:08:28.699 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: cat td/osd-scrub-repair/2/whoami 2019-12-21T16:08:30.409 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: '[' 1 = 1 ']' 2019-12-21T16:08:30.409 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph osd dump --format=json 2019-12-21T16:08:30.409 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: jq '.flags_set[]' 2019-12-21T16:08:30.409 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: grep -q '"noup"' 2019-12-21T16:08:30.409 INFO:tasks.workunit.client.0.smithi172.stderr:119575: 2019-12-21T16:08:28.672+0000 7f3ca7e45ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T16:08:30.410 INFO:tasks.workunit.client.0.smithi172.stderr:119575: 2019-12-21T16:08:28.674+0000 7f3ca7e45ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T16:08:30.410 INFO:tasks.workunit.client.0.smithi172.stderr:119575: 2019-12-21T16:08:28.676+0000 7f3ca7e45ec0 -1 WARNING: all dangerous and experimental features are enabled. 2019-12-21T16:08:30.410 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: wait_for_osd up 1 2019-12-21T16:08:30.410 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:914: wait_for_osd: local state=up 2019-12-21T16:08:30.410 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local id=1 2019-12-21T16:08:30.410 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: status=1 2019-12-21T16:08:30.410 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i=0 )) 2019-12-21T16:08:30.411 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i < 300 )) 2019-12-21T16:08:30.411 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: echo 0 2019-12-21T16:08:30.411 INFO:tasks.workunit.client.0.smithi172.stderr:119575: 0 2019-12-21T16:08:30.411 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: ceph osd dump 2019-12-21T16:08:30.411 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: grep 'osd.1 up' 2019-12-21T16:08:30.411 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: sleep 1 2019-12-21T16:08:30.411 INFO:tasks.workunit.client.0.smithi172.stderr:119575: 2019-12-21T16:08:29.195+0000 7f3ca7e45ec0 -1 Falling back to public interface 2019-12-21T16:08:30.412 INFO:tasks.workunit.client.0.smithi172.stderr:119575: 2019-12-21T16:08:30.134+0000 7f3ca7e45ec0 -1 osd.1 57 log_to_monitors {default=true} 2019-12-21T16:08:30.412 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i++ )) 2019-12-21T16:08:30.412 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: (( i < 300 )) 2019-12-21T16:08:30.412 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: echo 1 2019-12-21T16:08:30.412 INFO:tasks.workunit.client.0.smithi172.stderr:119575: 1 2019-12-21T16:08:30.412 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: ceph osd dump 2019-12-21T16:08:30.412 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: grep 'osd.1 up' 2019-12-21T16:08:30.413 INFO:tasks.workunit.client.0.smithi172.stderr:119575: osd.1 up in weight 1 up_from 60 up_thru 53 down_at 58 last_clean_interval [50,57) [v2:127.0.0.1:6810/171900,v1:127.0.0.1:6811/171900] [v2:127.0.0.1:6812/171900,v1:127.0.0.1:6813/171900] exists,up 356c69ab-d874-4bb4-813b-1608e502f8c0 2019-12-21T16:08:30.413 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=0 2019-12-21T16:08:30.413 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: break 2019-12-21T16:08:30.413 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: return 0 2019-12-21T16:08:30.413 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1239: objectstore_tool: wait_for_clean 2019-12-21T16:08:30.413 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1548: wait_for_clean: local cmd= 2019-12-21T16:08:30.413 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1549: wait_for_clean: local num_active_clean=-1 2019-12-21T16:08:30.414 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1550: wait_for_clean: local cur_active_clean 2019-12-21T16:08:30.414 INFO:tasks.workunit.client.0.smithi172.stderr:119575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1551: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2019-12-21T16:08:30.414 INFO:tasks.workunit.client.0.smithi172.stderr:119575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1551: wait_for_clean: get_timeout_delays 90 .1 2019-12-21T16:08:30.414 INFO:tasks.workunit.client.0.smithi172.stderr:119575: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1494: get_timeout_delays: shopt -q -o xtrace 2019-12-21T16:08:30.414 INFO:tasks.workunit.client.0.smithi172.stderr:119575: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1494: get_timeout_delays: echo true 2019-12-21T16:08:30.565 DEBUG:teuthology.exit:Got signal 15; running 2 handlers... 2019-12-21T16:08:30.565 DEBUG:teuthology.task.console_log:Killing console logger for smithi172 2019-12-21T16:08:30.565 DEBUG:teuthology.task.console_log:Killing console logger for smithi172 2019-12-21T16:08:30.566 DEBUG:teuthology.exit:Finished running handlers